Contents

• The Health Care Quality Problem
• Conceptualizing Health Care Quality
• Overview of Quality Measurement
• Development of Quality Measures
• Hospital Measures
• Physician Measures
• Plan Measures
• The Role of the Centers for Medicare and Medicaid Services (CMS)
• The Role of the Agency for Healthcare Research and Quality (AHRQ)
• Endorsement of Quality Measures
• National Quality Forum (NQF)
• The Role of AQA Adoption
• Selection and Implementation of Quality Measures
• Physician Quality Reporting Initiative (PQRI)
• Reporting Hospital Quality Data for Annual Payment Update (RHQDAPU)
• Selected Issues
• Policy Issues
• Technical Issues
• Conclusion

Summary

Problems with quality of health care in the United States earned the attention of the public with, and have steadily gained in importance since, the release of the first in a series of Institute of Medicine (IOM) reports on this topic, "To Err is Human: Building A Safer Health System," in 1999. The release of the IOM report represented a changing approach to addressing suboptimal health care quality, from one focused primarily on quality assurance to one focused on quality improvement broadly through the realignment of systems of delivering and financing care to incentivize quality of care. This issue now occupies a prominent position on the national agenda for health care reform and is the subject of considerable congressional interest.

Policy makers and others have offered a wide range of policy options to improve health care quality. This report provides information on the issue of quality measurement, which serves as the basis of several of the options policy makers are offering for addressing this problem. Specifically, recently introduced health reform bills have included a strong focus on supporting and expanding a coordinated national quality-measurement infrastructure. These include efforts that would broadly support the development of national priorities in the area of performance measurement; the development of new quality measures and identification of gaps in existing measures; the expansion of the endorsement process for quality measures; and the delineation of a process, in rulemaking, for selecting and implementing new quality measures for use in public health care programs.

This report begins with a brief description of health care quality activity conceptually, the types of quality information that support these activities, and how quality measurement can be placed within these frameworks. It provides a discussion of the definition of quality measurement and a description of the life cycle of quality measures, including the development of quality measures (across provider types), the process of endorsement of quality measures, and the process for implementing those measures. Overviews of selected examples of implementation, including the Physician Quality Reporting Initiative (PQRI) and the Reporting Hospital Quality Data for Annual Payment Update (RHQDAPU), are included. The report concludes with an overview of key technical and policy issues in quality measurement.

---

Measuring Health Care Quality: Measure Development, Endorsement, and Implementation

Problems with quality of health care in the United States earned the attention of the public with, and have steadily gained in importance since, the release of the first in a series of Institute of Medicine (IOM) reports on this topic, "To Err is Human: Building A Safer Health System," in 1999. The release of the IOM report represented a changing approach to addressing suboptimal health care quality, from one focused primarily on quality assurance to one focused on quality improvement broadly through the realignment of systems of delivering and financing care to incentivize quality of care. This issue now occupies a prominent position on the national agenda for health care reform and is the subject of considerable congressional interest.

Policy makers and others have offered a wide range of policy options to improve health care quality. This report provides information on the issue of quality measurement, which serves as the basis of several of the options policy makers are offering for addressing this problem. Specifically, recently introduced health reform bills have included a strong focus on supporting and expanding a coordinated national quality measurement infrastructure. These include efforts that would broadly support the development of national priorities in the area of performance measurement; the development of new quality measures and identification of gaps in existing measures; the expansion of the endorsement process for quality measures; and the delineation of a process, in rulemaking, for selecting and implementing new quality measures for use in public health care programs.

This report begins with a brief description of health care quality activity conceptually, the types of quality information that support these activities, and how quality measurement can be placed within these frameworks. It provides a discussion of the definition of quality measurement and a description of the life cycle of quality measures, including the development of quality measures (across provider types); the process of endorsement of quality measures; and the process for implementing those measures. Overviews of selected examples of implementation, including the Physician Quality Reporting Initiative (PQRI) and the Reporting Hospital Quality Data for Annual Payment Update (RHQDAPU), are included. The report concludes with an overview of key technical and policy issues in quality measurement.

The Health Care Quality Problem

The U.S. health care system is characterized by systemic quality shortcomings. The IOM stated in 2001 that "[c]rucial reports from disciplined review bodies document the scale and gravity of the problems. Quality problems are everywhere, affecting many patients. Between the health care we have and the care we could have lies not just a gap, but a chasm."¹ Significant activity has been undertaken to improve the quality of care delivered in the United States over the past decade, but a number of concerning trends remain.

The U.S health care system's quality problems are evidenced by a broad range of relevant quality indicators. These include high rates of medical and medication error; a high amenable mortality rate; overuse, underuse, and misuse of health care services; and a lack of, and variation in, diffusion of evidence-based guidelines.

In its 1999 study, the IOM reported that between 44,000 and 98,000 people die each year because of preventable medical errors, at a cost of between $17 billion and $29 billion per year. Serious adverse medication events are estimated to occur in up to 15% of hospitalized patients, and more than 100,000 deaths are attributed annually to such reactions.² A July 2008 study reports that despite spending two times more per capita than any other major industrialized country on health care, the United States has an amenable mortality rate—deaths that are amenable to health care intervention and may have been prevented by that intervention—of 110 per 100,000 (ranking 19^th out of 19 surveyed countries).³

With respect to the problems of overuse, misuse, and underuse of health care services, a study conducted by the Midwest Business Group on Health in 2003 found that approximately "30 percent of all direct health care outlays are the result of poor-quality care, consisting primarily of overuse, misuse and waste."⁴ Another study found that as many as 20% to 30% of patients received contraindicated care.⁵ In addition, a 2007 RAND study found that only 46.5% of children receive care recommended by evidence-based guidelines, and a similar RAND study conducted in 2003 concluded that adults receive only 55% of indicated care.⁶

According to the IOM, an average of 17 years is required for findings from randomized clinical trials to be implemented into clinical practice, and when they are, it is done so in an uneven fashion.⁷ Taken together, these findings evidence significant shortfalls in the quality of care provided in the United States.

Conceptualizing Health Care Quality

Despite the fact that most experts agree that health care quality is a serious problem deserving of attention, a precise operational definition for the concept can be difficult to identify. Although there is no single universally agreed upon definition for quality of care, or health care quality, perhaps the most commonly cited definition is the IOM's definition, as follows: "(t)he degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge."⁸

The 2001 IOM report "Crossing the Quality Chasm" further defines quality of care in terms of six domains:

• Effective. Providing services based on scientific knowledge to all who could benefit.
• Efficient. Avoiding waste, including waste of equipment, supplies, ideas, and energy.
• Equitable. Providing care that does not vary in quality because of personal characteristics.
• Patient-centered. Providing care that is responsive to and respectful of individual patient preferences.
• Safe. Avoiding injuries to patients from the care that is supposed to help them.
• Timely. Reducing waits and sometimes-harmful delays.

Most health care quality efforts aim to improve one or more of these six domains of quality of care, and quality care is often colloquially defined as simply delivering the right care to the right person at the right time. However basic it may appear, operationalizing these six domains, and thus improving health care quality, has proven difficult.

The effort to improve health care quality is operationally embodied by a wide range of quality activities. One way to consider these activities is by grouping them into one or more of the following four areas: (1) research (basic, translational, and health services); (2) quality measurement; (3) quality reporting (encompassing both increased transparency and incentive alignment); and (4) patient safety activities.⁹ Reliable and sound quality information plays a fundamental role in all of these health care quality activities, which may both use and generate such information. For those activities that rely on quality information, such as quality reporting, their effectiveness is directly dependent upon the quality of the information available. There are various types of quality information, including evidence syntheses, health technology assessments, clinical comparative effectiveness data, clinical practice guidelines, quality measure data, and medical error/near miss reports. In many cases, these different types of information are interrelated. For example, evidence syntheses are utilized as the basis for the development of clinical practice guidelines, which in many cases in turn serve as the basis for developing quality measures.

Quality information may be broadly leveraged through the types of activities referenced above for a number of purposes, including to educate and inform providers about performance; for public reporting to inform consumer decision making; to create payment incentives; to inform accreditation or maintenance of certification; to inform internal quality assurance or improvement efforts; for public health surveillance efforts and activities; and to inform system-level changes to improve safety. Quality information may be fed back to inform the modification of quality measures, incentive programs, quality assurance activities, health care provider education, or the processes of delivering health care with the goal of improving quality.

This report focuses specifically on quality measurement, one subset of quality activities, and will focus on the processes used to develop, endorse, select, and implement quality measures.

Overview of Quality Measurement

Broadly, quality measurement focuses on assessing processes, structures, and outcomes of health care, and for this reason, quality measures are informally categorized as structural, process, or outcome measures. This model of quality measurement rests on the premise that a sound structure will support appropriate processes of care that in turn will result in good outcomes.¹⁰ Ideally, process measures are supported by research demonstrating a link to improved outcomes, and all measures will assess things that are under the direct control of the health care system.

Structural measures assess whether "the resources and organizational arrangements are in place to deliver care," for example, staffing ratios or health information technology (HIT) implementation. Process measures assess whether the "appropriate provider activities are carried out to deliver care," for example, percentage of indicated patients receiving mammograms within a given timeframe. Finally, outcomes measures assess "the results of physician and other provider activities," for example, 30-day mortality rates post-hospitalization for pneumonia.¹¹ Composite measures, which summarize performance across separate and discrete measures of quality, are a useful tool for certain end-users, such as policy makers and the general public.¹² The Centers for Medicare and Medicaid Services (CMS) now includes several Agency for Healthcare Research and Quality (AHRQ)-developed composite measures in its Reporting Hospital Quality Data for Annual Payment Update (RHQDAPU) program (described later in this report).

In addition, quality measures may be developed in order to collect patients' reports of system performance. An example of this type of measure may be found in the Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey and the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey, both funded by AHRQ. While these measures may assess the delivery of evidence- or consensus-based clinical care, they also include an evaluation of the human aspects of how that care was delivered.

Quality measures may attempt to assess or evaluate one of three broader breakdowns in health care quality: (1) overuse of health care services, (2) misuse of health care services, or (3) underuse of health care services. This approach encompasses variation in care patterns as well as disparities in quality of care received. The development of quality measures may be based either on existing clinical practice guidelines (evidence-based) or on consensus-based guidelines. Generally, utilizing an evidence-based guideline as a basis for a quality measure is preferred over using a consensus-based one; however, as there are significant gaps in the evidence base, in many cases, consensus-based guidelines are the only option available. Although the development of reliable and valid quality measures tends to be challenging, the development of structural and process quality measures may be less complicated than the development and specification of outcomes measures, which requires complex risk-adjustment manipulations to account for variation in patient case-mix. Data sources for quality measures vary, but include four primary sources: (1) administrative claims data, (2) medical records, (3) medical registries, and (4) patients' reports.

Quality measures may be used for a number of purposes. They may be used to assess performance at the level of a hospital (or other health care facility, for example, a nursing home or outpatient surgical center); a health care provider, group of providers, or practice group; or a health plan. They may be used by payers, purchasers, health care consumers, regulators, accrediting or licensure bodies, researchers, health care providers, or health care facilities. Finally, quality measures may be used as the basis for policies that realign payment incentives; to facilitate internal quality improvement or quality assurance efforts; to empower consumer decision making; or to provide feedback on performance—for the purposes of self-improvement— to health care providers, health plans, or health care facilities.

The life cycle of a quality measure is composed of three basic components: the creation and specification of the measure, the endorsement of the measure, and the selection and implementation of the measure. This process may not always be perfectly linear in all cases (for example, an entity may adapt an existing endorsed measure for use in one of its quality-related activities), but with respect to the creation of de novo quality measures, these are the ordered steps generally followed. Each stage is discussed in more detail in the following sections.

There is a wide range of current efforts to develop quality metrics that may be used in performance measurement activities and, likewise, there are a number of different entities involved in the development, endorsement, selection, and implementation of quality measures. AHRQ reports in its 2008 National Healthcare Quality Report that the "often opportunistic, incremental, and fragmented development of quality measures without detailed consideration of data sources, analysis and maintenance requirements, and user needs" is a challenge, and that "uncoordinated and isolated measure development can lead different groups to create and advocate competing and sometimes conflicting measures of the same process or outcome."¹³

In an attempt to better coordinate performance measurement and other quality activities, in November of 2008, the National Quality Forum (NQF) convened the National Priorities Partnership (NPP), a collaborative effort between 28 key health care stakeholders representing all major health care perspectives.¹⁴ The goal of the NPP is to rapidly improve quality in six priority areas, including patient and family engagement, population health, safety, care coordination, overuse, and palliative and end-of-life care. The NPP aims to achieve measurable progress in these six areas, specifically focusing on payment, public reporting, quality improvement, and consumer engagement.

The Medicare Improvements for Patients and Providers Act of 2008 (MIPPA, P.L. 110-275) further emphasized the importance of "an integrated national strategy and priorities for health care performance measurement in all applicable settings."¹⁵ MIPPA required the Secretary to enter into a contract with a consensus-based entity (and provided as an example the National Quality Forum) to help strengthen and better focus the nation's performance-measurement infrastructure, and specifically to synthesize evidence and convene stakeholders to provide recommendations on a national performance-measurement infrastructure; to endorse and maintain measures; and to report annually to the Secretary on these activities. The National Quality Forum was awarded this contract and recently released its first report, Improving Healthcare Performance: Setting Priorities and Enhancing Measurement Capacity, in fulfillment of this statutory requirement.¹⁶

Development of Quality Measures

The initial phase in the life cycle of a quality measure is the development stage. In this stage, measures are conceived of, created, specified, refined, and oftentimes field tested. Quality measures are developed by a number of different measure developers, including The Joint Commission, the National Committee for Quality Assurance (NCQA), and the American Medical Association (AMA)-convened Physician Consortium for Performance Improvement (PCPI), for application in a wide variety of settings, including hospitals, nursing homes, clinics, and to evaluate health plans.

Measures may generally be developed in one of two ways: they may be created entirely anew, or de novo, or existing measures may be identified and adapted for a specific purpose (i.e., hospital accreditation). Traditionally, developers have focused on measures of process rather than outcomes, and they generally direct their efforts toward developing measures focused on a specific setting (e.g., hospital) or in a particular clinical area (e.g., pneumonia). Gaps have occurred in measure development where services span provider settings or where no single entity has sole responsibility for providing a service (e.g., coordination of care or end-of-life care). In addition, measure development may be guided by overuse or cost intensiveness, or rather, may be guided by explicit links to improved outcomes, regardless of cost or cost burden. From a value-based purchasing perspective, development would ideally be guided by both; however, from the consumer perspective, outcomes may be more important than costs.

This section details the processes used by measure developers to develop quality measures for evaluating the performance of hospitals, physicians, and health plans. An examination of these processes reveals that they share several attributes: they all solicit public input on measures, they all rely on panels or workgroups of technical and/or clinical specialists for guidance, and they all subject measures to some degree of field testing. In addition, both the Joint Commission and NCQA utilize a set of clearly delineated and specific attributes by which to assess and evaluate measures' value and thus guide their development activities. NQF has recognized the importance of testing quality measures by instituting a mechanism of "time-limited" endorsement for a period of 12-24 months for those measures that meet all of NQF's standards for endorsement, but that have not been subjected to rigorous field testing.¹⁷

Hospital Measures

The Joint Commission has one of the most well-developed and clearly specified processes of quality-measure development, and has been involved in developing performance measures since 1986 for hospitals (these measures may be used by other health care organizations, including home care, behavioral health care and long term care organizations). Reporting quality data was made a formal requirement for Joint Commission hospital accreditation in 1998.

The Joint Commission, formerly the Joint Commission on the Accreditation of Healthcare Organizations, or JCAHO, is a private, non-profit organization that establishes quality and safety standards and accredits health care organizations for participation in federal health insurance programs. For example, The Joint Commission accredits approximately 80% of hospitals for participation in Medicare by assessing their compliance with standards related to patient rights and education, infection control, preventing medical errors, medical staffing ratios, and data-collection processes. The organization describes its mission as "continuously improving the safety and quality of care provided to the public through the provision of health care accreditation and related services that support performance improvement in health care organizations."¹⁸

When The Joint Commission began its work in the area of quality measurement in 1986, it focused on developing quality measures completely de novo, an involved and resource-intensive endeavor. However, while The Joint Commission was engaged in this work, many other measure developers began developing and implementing their own measures. For this reason, The Joint Commission changed course in the mid-1990s and redirected its efforts toward adapting (evaluating and testing) existing quality measures for its purposes. This shift resulted in the creation of a standardized process for adapting existing measures consisting of a series of specific steps. The Joint Commission focuses on developing its measures in sets, and defines a "core measure set" as "a unique grouping of performance measures, carefully selected to provide, when viewed together, a robust picture of the care provided in a given focus area."¹⁹

Initial measure development requires significant preliminary work, including a review of the literature to evaluate the evidence base and identify existing measures in a particular topical area. The findings from this review are summarized and serve as the basis of a measurement framework, which is used throughout the development process. The first formal step in the process is the empanelling of a group of experts with balance as a key focus. This expert advisory panel includes representation from relevant medical, nursing, and pharmacy professionals, as well as state hospital associations, consumers, and CMS. Funders are not represented on these panels to avoid conflicts of interest, and full disclosure is required of all members.

After the panel has been convened, The Joint Commission issues a "call for measures" to solicit measures from all known measure developers. The measures received are vetted by staff, and evaluated against The Joint Commission's attributes of core performance measures and associated evaluation criteria. These criteria include the following:

• Measures target improvement in the health of populations.
• Measures are precisely defined and specified.
• Measures are reliable.
• Measures are valid.
• Measures are easily interpreted by users.
• Measures are risk-adjusted or stratified.
• Measures are under provider control.
• Measures have publicly available measure constructs.
• Measures are useful in the accreditation process.
• Measures rely on accessible data and low-cost data collection efforts.

There is significant overlap between The Joint Commission's measure attributes, NQF's measure evaluation criteria, and NCQA's desirable measure attributes.²⁰

Measures that meet these criteria are forwarded to the expert advisory panel, and final candidate measures are posted to The Joint Commission's website, where they undergo a 30-day comment period. Measures are adjusted according to the feedback received during the comment period, at which point general specifications may be formulated and field testing begun. The Joint Commission's field testing is well developed, and involves both alpha testing—for feasibility and utility—and beta testing—for reliability, validity, and cost, among other things. There are opportunities for measure adjustment and refinement at all points in this process. Based on the results of the field testing, the expert advisory panel recommends implementation of a set of final measures, and the measures will then be integrated into ORYX, the performance measurement component of The Joint Commission's accreditation process.

Physician Measures

The American Medical Association (AMA)-convened Physician Consortium for Performance Improvement (PCPI) is a physician-led, multi-stakeholder group that has taken the lead in the development of quality measures for physicians. The Consortium, staffed by the AMA and led by a 23-member Executive Board, is composed of more than 100 national and state medical societies, experts in methodology and data collection, AHRQ, and CMS. The PCPI supports, and provides in-kind resources for, the establishment of topic-specific work groups that include both clinician specialists as well as technical methodological experts to develop disease- or condition specific-measures.

To date, the PCPI has developed more than 250 measures in over 40 clinical areas, and the measures are heavily used by CMS's Physician Quality Reporting Initiative (PQRI). In addition, uses for these measures include integration into electronic health records (EHRs), Maintenance of Certification (MOC) for physicians, and CMS demonstration projects, and measures may be used for quality improvement purposes, as well as for accountability.

The process used by PCPI to develop measures is not as longstanding as are those employed by The Joint Commission and NCQA, simply by virtue of the fact that the organization is far younger. While The Joint Commission began developing hospital performance measures in 1986, and NCQA began developing its HEDIS measures in 1992, the PCPI was established in 2000 to help guide medical specialty groups in generating physician-level measures, at least partially in anticipation of a formal, statutorily established physician quality reporting system for Medicare-participating physicians. The scope and depth of PCPI's work was described in late 2007 as nascent;²¹ however, PCPI has had the advantage of being able to draw on existing work in this area, and quickly developed its own robust process for measure development (see Figure A-2).

PCPI has been endeavoring to make strategic changes in emphasis as it matures. It has refocused its measure-development efforts, and requires the following three criteria be met by any topic prior to proceeding with development in the area:

• 1. The topic is an area designated as high impact (by the IOM, NPP, etc.).
• 2. The topic is a gap area or an area with high variation in care.
• 3. The topic has an adequate evidence base.

If these three criteria are met, PCPI evaluates whether the topic under consideration is likely to generate measures in the following four areas, which it terms "high value": care coordination, patient safety, appropriateness/overuse, and quality improvement collaboratives. The Consortium has also shifted its focus to developing a comprehensive "core" measure set.

After a topic has been approved, an topic-specific expert work group is convened, consisting of two co-chairs with expertise in the clinical area as well as in measure-development methodology; clinicians in all relevant areas; methodologists; guideline experts; patients, purchasers, consumer groups and health plans; and sometimes coding experts, depending on the topical area. Importantly, representatives from The Joint Commission and NCQA are included in the composition of the work groups to ensure measure harmonization to the greatest extent possible. Prior to beginning its actual measure drafting, the work group conducts a literature review (and review of the existing evidence base) and identifies desired outcomes (with respect to either a certain condition or procedure). The work group meets once in person and conducts several conference calls to facilitate the measure-drafting process, after which public comment is solicited on the measures. The work group will then review the comments and incorporate them as necessary. The revised measures are then submitted for approval to PCPI. Measure testing is undertaken on the measures to assess both reliability and feasibility, and then measures are submitted to NQF for potential endorsement and also released for public use.

Plan Measures

The National Committee for Quality Assurance (NCQA), a private, 501(c)(3) not-for-profit organization dedicated to improving health care quality through the accreditation of health plans and quality-measure development, has taken the lead role in the development of quality measures used to evaluate health plans. Its programs, which focus on evaluating the performance of health plans, also include the development of report cards, public-policy documents, and educational activities.

Perhaps NCQA's most well-known performance evaluation tool is the Healthcare Effectiveness Data and Information Set (HEDIS), a set of standardized measures used by more than 90% of health plans to measure clinical performance in areas such as medication use, control of high blood pressure, breast cancer screening, immunization, and comprehensive diabetes care, among others. HEDIS 2009 consists of quality measures in eight health care domains, and may be used to assess clinical quality in both managed care organizations (MCOs) and preferred-provider organizations (PPOs).

HEDIS measures are developed by NCQA using a process involving input from a broad range of stakeholders, including purchasers, policy makers, health care providers, patients, and health plans (see Figure A-3). There are several key groups involved in this process: Technical Advisory Groups (TAGs) provide guidance on methodological considerations; the Committee on Performance Measurement (CPM) provides guidance throughout the measure-development process and makes final decisions regarding adoption; and Measurement Advisory Panels (MAPs) report to the CPM and provide specialized guidance for measure development in a specific area.

The process begins with the identification of a topic, a literature review, and measure identification. Measures are evaluated for inclusion in the development process against a series of three key criteria: relevance, scientific soundness, and feasibility. A MAP is created to guide development, and initial measure specifications are drafted by the TAG and then presented to the MAP. Measures are then field tested, revisions are made, and then the measures are presented to the CPM. The revised measures are posted on the NCQA website for a period of 30 days, with a request for public comment. Finally, the CPM reviews all comments, revises the measures as needed based on the comments, and then votes formally to adopt the measures for inclusion in HEDIS.

The Role of the Centers for Medicare and Medicaid Services (CMS)

The Centers for Medicare and Medicaid Services (CMS) plays an important role in providing overall guidance for the development of measures for use in its multiple quality activities. This is reflected in an agency document entitled Quality Measures Development Overview, which explains that "CMS has developed a standardized approach for the development and maintenance of quality measures it uses in its various quality initiatives and programs."²² This process, referred to by CMS as the Measures Management System and described in more detail below, involves participation by three main entities: CMS, a measure developer (e.g., PCPI or NCQA), and a Technical Expert Panel (TEP) whose responsibility it is to provide expert guidance to both CMS and the measure developer (see Figure A-1).

The measure-development process CMS has outlined is followed by the CMS-funded measure developer and shares many similarities with those employed by The Joint Commission, NCQA, and the PCPI. It begins with the definition of a topic and the empanelling of a topic-specific TEP. With the guidance of the TEP, the measure developer develops a measure framework, identifies candidate measures, and provides these measures, accompanied by general information, to the TEP for review. After the TEP reviews the candidate measures list, it is submitted to CMS for review and approval, and public comment may be solicited. Measures approved by CMS at this stage move forward to the specification-development stage. Once specifications have been created for the measures, they are approved by CMS, the measures are tested for both reliability and feasibility, and public comment is solicited. Refinements may be made to the measures at this point, based on public feedback, and they will go through a final approval process and then be submitted for consensus endorsement (e.g., to the NQF).

Although CMS relies on both TEPs and measure developers in the process of developing the quality measures it will employ in its quality programs, the agency itself plays a critical role at many points during this process. This distinction is not clearly delineated by the agency. In its recently released Roadmap for Quality Measurement in the Traditional Medicare Fee-for-Service Program, CMS states that "(i)n general, CMS does not play a significant role in the development or endorsement of quality measures."²³ This statement, in the context of the Quality Measures Overview Document, may understate the role of CMS in this process.

While CMS does not carry out the technical process of specifying quality measures (this task falls to the measure contractor), it is a critical part of the process at many points. CMS helps to define the topic for which measures will be developed; provides approval of candidate measures for specification; determines whether public comment will be required in the early stages of measure development (pre-specification); approves measure specifications; determines whether measures with specifications will be submitted for consensus review prior to testing; and gives final approval of measures for use in its programs after completion of testing and formal public comment. The agency plays a guiding role in the process and appears to make the final decisions regarding measure advancement.

The Role of the Agency for Healthcare Research and Quality (AHRQ)

The Agency for Healthcare Research and Quality (AHRQ) plays an important role in the development of quality measures. The agency's authorizing language stipulates that it shall conduct and support research, evaluations, and training; support demonstration projects, research networks, and multidisciplinary centers; provide technical assistance; and disseminate information on health care and on systems for the delivery of such care. This activity specifically targets the following areas:

• the quality, effectiveness, efficiency, appropriateness and value of health care services;
• quality measurement and improvement;
• the outcomes, cost, cost-effectiveness, and use of health care services and access to such services;
• clinical practice, including primary care and practice-oriented research;
• health care technologies, facilities, and equipment;
• health care costs, productivity, organization, and market forces;
• health promotion and disease prevention, including clinical preventive services;
• health statistics, surveys, database development, and epidemiology; and
• medical liability."²⁴

AHRQ carries out a number of measure-development activities, including development of the Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey, the AHRQ Quality Indicators, and the National Healthcare Quality and National Healthcare Disparities reports. The measures developed under these various activities are used both to track and report on quality at the national level (the National Healthcare Quality and National Healthcare Disparities reports) and to monitor quality at the institution level (CAHPS and the Quality Indicators).

The AHRQ Quality Indicators is a family of measurement indicators, including indicators focusing on prevention, inpatient care, patient safety, and pediatric care, that make use of hospital inpatient administrative data. For 2009, CMS has incorporated some of AHRQ's Patient Safety Indicators (PSIs) and Inpatient Quality Indicators (IQIs) into its Reporting Hospital Quality Data for Annual Payment Update (RHQDAPU) program. In addition, a modified version of the CAHPS survey, the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey, is used by CMS in its RHQDAPU program. This information is also publicly reported on CMS's Hospital Compare website.

Endorsement of Quality Measures

The National Quality Forum (NQF) is the entity in the United States with the lead responsibility for endorsing quality measures. Numerous measure developers, including NCQA, PCPI, and The Joint Commission, submit their measures to NQF for endorsement, and other entities, including CMS, the Hospital Quality Alliance (HQA), medical boards, and health care providers, often choose quality measures for their quality activities from among the list of NQF-endorsed measures. The endorsement process provides a mechanism for recognizing those measures that meet certain standards and allows interested parties to identify easily a set of valid and reliable measures. This is valuable, given the large number of extant quality metrics.²⁵

Many entities rely on NQF-endorsed measures for use in their quality activities. For example, CMS's Physician Quality Reporting Initiative (PQRI) is statutorily required in 2008 and 2009 to use measures that have been "adopted or endorsed by a consensus organization (such as the National Quality Forum or AQA)"²⁶ and is required in 2010 and subsequent years to use "such measures selected by the Secretary from measures that have been endorsed by the entity with a contract with the Secretary under section 1890(a) of the Social Security Act."²⁷ Also, with respect to CMS's Reporting Hospital Quality Data for Annual Payment Update (RHQDAPU) program, the Secretary is required to "add other measures that reflect consensus among affected parties and, to the extent feasible and practicable, shall include measures set forth by one or more national consensus building entities."²⁸

National Quality Forum (NQF)

The National Quality Forum (NQF), a public-private not-for-profit membership organization with more than 350 members, was established in 1999 with the mission of improving the quality of health care, specifically through setting national goals for improvement, through endorsing quality measures, and through education and outreach to facilitate the realization of the quality goals it has recommended. The President's Advisory Commission on Consumer Protection and Quality in the Health Care Industry (the Quality Commission) recommended the establishment of what would later come to be known as NQF in a March 1998 report entitled "Quality First: Better Health Care for All Americans." This report recommended "establishing two complementary entities, one public and one private, to provide ongoing national leadership in health care quality improvement. The Commission recommends the creation of a broadly represented, publicly administered—Advisory Council for Health Care Quality—and a privately administered—Forum for Health Care Quality Measurement and Reporting."

The NQF is a voluntary consensus standards-setting organization, as defined in law by the National Technology Transfer Advancement Act (NTTAA) and by the Office of Management and Budget (OMB) Circular A-119. According to OMB Circular A-119, the five attributes of a voluntary consensus standards-setting body are balance, openness, due process, consensus, and having an appeals process.²⁹ Currently, NQF is the only body that meets these formal criterion as a voluntary consensus standards-setting body for health quality measures. In addition, the NTTAA requires federal agencies to use voluntary consensus standards in their activities, unless doing so conflicts with existing law or is impractical.³⁰

NQF's most prominent and visible role has been in its capacity as a national-level quality measure endorser (it also endorses frameworks, reporting practices, and preferred practices), and it has to date endorsed more than 500 quality measures. Its decisions are highly respected and valued by all stakeholders; however, because the endorsement process is laborious and methodical, and tends to be lengthy, measure endorsement cannot keep pace with measure development. NQF has begun to emphasize the importance of harmonizing measures, and places value, for example, on setting neutrality and common conventions for measures related to a specific condition or targeted group.

NQF utilizes a rigorous and methodical process by which it reviews and judges quality measures for potential endorsement, known formally as the consensus development process (CDP) (see Figure B-1). This process comprises seven formal steps: (1) the formation of a steering committee, (2) a call for measures, (3) measure evaluation, (4) public and member comment, (5) member voting, (6) consensus standards approval committee review, and (7) appeals.³¹ To begin the process, NQF puts out a public call for members for a steering committee. It is the job of the committee to guide the evaluation process and provide expert input and guidance. The next steps entail a call for measures, where measure developers (not necessarily NQF members) are asked to submit their measures for consideration for endorsement. Developers have a period of 30 days during which to submit their measures.

Once measures have been collected, the steering committee begins the evaluation process. Measures are evaluated against four criteria, including the following:

• whether the measure is linked to gains in quality and health outcomes,
• whether the measure is reliable and valid,
• whether the measure is usable (results are easy to understand and useful), and
• whether the measure is feasible (ease of access to required data).

A measure is initially evaluated against the first criterion—the relationship between a measure and quality and health outcomes—and if it is judged to pass this threshold, it will then be evaluated against the remaining three criteria. A measure may fail to gain endorsement, for example, if it is judged to assess a core competency, rather than high-quality care.

For measures that survive this evaluation process, there is a 30-day public comment period, after which the public's comments are reviewed by the steering committee and incorporated as appropriate. At this time, the revised measures are posted and NQF members have a 30-day period during which they are invited to vote either in favor of or in opposition to endorsement. The final endorsement recommendation is made by the Consensus Standards Approval Committee (CSAC) to the NQF Board of Directors, which affirms the CSAC's recommendation. Appeals may be made for a 30-day period following ratification of endorsement by the NQF Board of Directors.

The Role of AQA Adoption

The AQA Alliance, formerly known as the Ambulatory Care Quality Alliance, was established in 2004 and is co-sponsored by the Agency for Healthcare Research and Quality (AHRQ), the American College of Physicians (ACP), the American Academy of Family Physicians (AAFP), and America's Health Insurance Plans (AHIP). AQA is a consensus organization with broad stakeholder representation, which until recently "adopted" physician-quality measures, a status which served as AQA's recommendation that a measure is ready for implementation. During this process, AQA considered measures that were either NQF-endorsed, or would be going through the NQF-endorsement process, for adoption. If an AQA-adopted measure went through the NQF endorsement process but failed to gain endorsement, AQA adoption was rescinded. Although AQA focuses broadly on the three issues of performance measurement, data aggregation, and reporting, one of its key goals was to reach consensus on a set of physician-level quality measures that could be used by government purchasers or to guide private health insurance plan contracting decisions. This should help ultimately to standardize the performance measures used to assess and evaluate physician performance. AQA has now shifted its focus to identifying how quality measures are being used and implemented, as well as assessing where gaps exist in extant quality measures.

AQA-adopted measures may be used by CMS in its Physician Quality Reporting Initiative (PQRI), and many AQA-adopted measures were also used in PQRI's predecessor program, the Physician Voluntary Reporting Program (PVRP). For the 2009 PQRI, the measures meet one of the four following criteria:

• they are measures that were selected from the 2008 measure set,
• they are NQF-endorsed measures,
• they are AQA-adopted measures, or
• they were not NQF-endorsed or AQA-adopted at the time the proposed rule was published by CMS, but received endorsement or adoption by August 31, 2008.³²

This allowed for a mechanism whereby measures were able to be implemented by CMS's PQRI without NQF endorsement, since AQA is recognized as a "consensus organization per section 1848(k)(2)(B) of the (Social Security) Act."³³ Given the lengthiness of NQF's endorsement process, the large number of quality measures needing to go through the process, and AQA's recognized status as a consensus organization under law, AQA adoption allowed for expedited implementation of physician measures. As one commenter notes, this highlights a major challenge and tension inherent in the development and implementation of quality measures: "balancing the need for scientifically sound development processes against the need to have measures available for use."³⁴

Selection and Implementation of Quality Measures

Once quality measures have been developed and ideally received endorsement, they will be considered for use in various quality activities. Since quality activities have varying goals (e.g., accountability, quality improvement) and are conducted in different settings for a variety of purposes, the selection stage is critical. This stage is especially consequential if the measures are being used for purposes of accountability, so the processes used by CMS to select measures for its pay-for-reporting programs (as part of its value-based purchasing initiative), and by CMS and other stakeholders for public reporting, are of particular import. The criteria used by CMS to select measures for its PQRI and RHQDAPU programs are discussed in more detail below.

AQA has also played a prominent role in the measure selection stage for physician measures. The AQA encourages and facilitates health plans and physicians to collaborate in determining which quality measures developed by the PCPI or the NCQA should be recommended for implementation. These recommendations are directed primarily to CMS for measure inclusion in its PQRI program, but also to smaller private payers for use in pay-for-performance programs. In this way, AQA contributes to prioritizing measures for implementation.

The following sections describe details of selected examples of implementation, specifically the Physician Quality Reporting Initiative (PQRI) and the Reporting Hospital Quality Data for Annual Payment Update (RHQDAPU) Program.

Physician Quality Reporting Initiative (PQRI)

The establishment of a quality reporting system for physicians was required by section 101(b) of the Tax Relief and Health Care Act of 2006 (TRHCA, P.L. 109-432), which also outlined a process for arriving at eligible quality measures. In response, CMS established the Physician Quality Reporting Initiative (PQRI) and began collecting physician-level quality data in July of 2007. PQRI was modeled on the Physician Voluntary Reporting Program (PVRP), which enabled physicians to voluntarily report quality data to CMS with no payment incentives, allowing the agency to establish a sound reporting infrastructure.

Unlike the PVRP, PQRI is a pay-for-reporting system and includes financial incentives for participation. Section 101(c) of TRHCA provided for financial incentives for the PQRI program and allowed for the awarding of an incentive payment to physicians based on the reporting of certain quality measures to begin in 2007. PQRI was extended for 2008 and 2009 by the Medicare, Medicaid, and SCHIP Extension Act of 2007 (MMSEA, P.L. 110-173) and was made permanent by the Medicare Improvements for Patients and Providers Act of 2008 (MIPPA, P.L. 110-275), which also allowed for incentive payments to continue through 2010.

The 2009 PQRI measure set contains more than 150 measures, and physicians are awarded a bonus payment of 2.0% of total allowed Physician Fee Schedule charges for covered professional services if they meet specified quality-reporting requirements. Measure reporting is either claims based or registry based,³⁵ and physicians may choose to report either on specific measures groups designated by CMS or on a series of unrelated measures, of the physician's choosing, that apply to their patient population.³⁶ The quality information collected through the PQRI program is not available to the public.

As noted previously, the quality measures included in the 2009 PQRI measure set were measures that either were included in the 2008 set, were NQF-endorsed, were AQA-adopted, or were undergoing consideration by either AQA or NQF for either adoption or endorsement. CMS identified six criteria that it employed when considering a measure for inclusion in its 2009 PQRI program. These include whether measures

• satisfy statutory requirements for selection (i.e., the measure was developed using a consensus-based process);
• are functional (i.e., useable);
• increase opportunities for eligible professionals to participate in the program or apply to an area without applicable measures;
• align with other CMS program health care goals;
• support CMS priorities (e.g., prevention, chronic conditions, high-cost and high-volume conditions, improved care coordination); and
• address various aspects of clinical care, including process, outcome, structure or patient experience.³⁷

In addition, CMS takes the public comments received during the rulemaking process, and in response to other informal requests for input, into account when selecting measures for the PQRI program.

Reporting Hospital Quality Data for Annual Payment Update (RHQDAPU)

The development of the Reporting Hospital Quality Data for Annual Payment Update (RHQDAPU) program, as statutorily directed by Section 501(b) of the Medicare Prescription Drug, Improvement, and Modernization Act of 2003 (MMA, P.L. 108-173), represented one of CMS's early steps in the area of value-based purchasing. The RHQDAPU program requires participating hospitals to report specific quality data to CMS in order to receive a full annual payment update. Participation is voluntary, and hospitals may withdraw at any time.

The Deficit Reduction Act of 2005 (DRA, P.L. 109-171), Section 5001(a), provided for the continuation of this reporting requirement, and increased the at-risk percentage of the annual payment update from 0.4% to 2.0%. This section also required the Secretary to make the collected facility-specific quality data public.

The 2009 RHQDAPU measure set (for determining FY2010 payment) contains more than 40 measures and includes process measures; patient perspectives (through the HCAHPS, or Hospital Consumer Assessment of Healthcare Providers and Systems, survey); outcomes measures (30-day mortality rates); 30-day hospital readmission rates; selected AHRQ patient safety indicators (PSIs) and inpatient quality indicators (IQIs), three of which are composite measures; one measure assessing participation in a clinical registry; and one nursing-sensitive measure. The data reported and collected encompass several conditions identified as common in the Medicare population, including acute myocardial infarction, heart failure, and pneumonia, and also address surgical care improvement.

CMS has continually expanded the measure set used in its RHQDAPU program, building on the initial "starter set" of 10 measures, as directed by statute.³⁸ Initial expansion was guided by a baseline set of measures developed by the IOM and presented in its report "Performance Measurement: Accelerating Improvement." These measures included a number of Hospital Quality Alliance (HQA) measures, several structural measures, and the HCAHPS survey.³⁹ Established in 2002 by the Federation of American Hospitals (FAH), the American Association of Medical Colleges (AAMC), and the American Hospital Association (AHA), HQA has recommended hospital quality measures much as AQA adopted physician measures. HQA also collects and publishes quality information in partnership with CMS, although there is not perfect overlap between the HQA and RHQDAPU measure sets. The HQA has played an important role in adopting and implementing hospital quality measures, often adopting measures ahead of CMS and in this way allowing for informal user testing prior to adoption by CMS.

In addition to statutory requirements, the selection of measures for inclusion in the RHQDAPU program is guided by a series of six specific goals. These include

• expanding measures beyond process measures, to measures of outcome, patient perspectives, and efficiency;
• expanding the scope of hospital services to which the measures apply;
• considering the burden on hospitals;
• harmonizing the measures with other CMS quality programs;
• weighing the relevance and utility of the measures compared to the burden on hospitals; and
• using measures that are based on currently reported data (i.e., to clinical data registries or all-payer claims databases) or that do not require chart abstraction.⁴⁰

CMS also prioritizes measures that evaluate high-burden, high-volume and high-cost conditions in the Medicare population and conditions (with established clinical guidelines) that persist in demonstrating wide variation in treatment and cost.⁴¹ As it does when selecting measures for inclusion in PQRI, CMS also considers public comments it receives pursuant to the rulemaking process when selecting measures for the RHQDAPU program.

Selected Issues

The complexity of quality measurement, and the importance of its potential applications, has generated a number of policy and technical issues in terms of both measure development and implementation. Especially in the context of health care reform, these issues may be useful to consider as Congress develops policies aimed at improving health care quality. Some of these issues are discussed in more detail below.

Policy Issues

Perhaps the most commonly raised issue with respect to measuring, and therefore reporting, quality is the lack of harmonization of reporting requirements. Many reporting entities (physicians or hospitals, for example) report different quality measures to different entities for different purposes, and the burden and cost associated with this effort can be high. Efforts at streamlining requirements have been initiated, for example, by The Joint Commission and CMS; however, reporting requirements still tend to be complicated and laborious, as well as fairly costly. In addition, reporting entities may not always be provided with timely and useful feedback on their performance, decreasing the potential utility of the activity.

As noted previously in this report, a lack of standardized quality measures, and the lack of a uniform approach to generating these measures, have presented both challenges and opportunities. The diversity of stakeholders involved in the quality-measurement enterprise has resulted in a high degree of innovation in terms of measuring quality and related activities. It has also allowed practices to be developed that best suit a particular setting (e.g., hospitals) or purpose (e.g., accreditation). However, it has also created challenges with respect to the development of a set of measures that apply uniformly across all clinical areas and conditions, and this has resulted in a situation where measures may be unavailable to assess certain areas, or where only measures of questionable quality may be available. Measure gap areas include, for example, end-of-life care.

These measure gap areas may also result from a lack of evidence-based clinical guidelines in a particular clinical area or for a specific clinical condition. In these cases, in order to meet expedited policy goals, measures may be developed using consensus-based guidelines. While this has the benefit of facilitating measurement, it may also result in less robust quality measures in some cases.

Technical Issues

From a technical perspective, there are numerous issues that would ideally be addressed in order to achieve a robust quality measurement system. These issues may be divided into those that address validity and those that address reliability.

With respect to validity—that is, ensuring that quality measures are in fact measuring what they are intended to measure—a number of issues are critical. These include the proliferation of generally similar, yet slightly different, quality measures that could be used to measure the same process, structure, or outcome. For example, a search of AHRQ's National Quality Measures Clearinghouse (NQMC) identifies 10 quality measures related to receiving mammograms. This not only increases the burden of reporting on reporting entities, but also creates challenges with respect to drawing valid comparisons between measures. The problem of multiple similar measures is partially mitigated both by endorsement of measures and by the NQMC itself, which provides a uniform mechanism for comparing measures.

Issues with respect to accurate attribution—assigning the provision of a service to a specific physician or group practice—can make assessing quality of care challenging. Patients see multiple providers, which can complicate attributing processes to either a single encounter or provider. Issues of attribution may be alleviated, at least in part, by collecting patients' reports on the care they receive.

Finally, it can be difficult to link performance on outcome measures directly to the actions of providers. Health outcomes are influenced by a variety of variables that are not under the direct control of either the health care provider or the health care system, including environmental factors, access to health care services, and genetics. In addition, even though process and outcome are generally accepted to have a tighter linkage, issues have recently been raised regarding this relationship as well. In particular, it has been unclear whether strong performance on process of care measures is linked to decreased mortality rates. There is some evidence that in fact this relationship is not as tightly linked as it was believed to be.⁴²

Ensuring reliability—that is, that measured results are consistent across repeated measurements—can also be technically challenging. Data sources must be appropriate and complete, as well as comparable across settings. Sample size (in this case, opportunities to provide "measured" care or to treat specified conditions for which outcomes are being measured) must be large enough to allow for drawing valid inferences about the performance of a single organization, a single provider within an organization, or a particular racial or ethnic group. Finally, standardization of quality measures is necessary in order to ensure that the same thing is being measured, according to the same specifications, across measurements.

Conclusion

The issue of improving the quality of health care is multifaceted, and policy makers have offered numerous approaches to addressing this policy problem. Measuring the quality of care delivered has served as the basis for a number of policy options being considered, but the effectiveness of these policy approaches depends on the quality of the data generated through measurement activities. The quality of this data in turn depends on the quality of the measure development process, as well as the way in which the measures are used and for what purpose. Congress may want to consider the issues outlined in this report when developing policy options that rely on quality measurement.

Appendix A. Quality Measure Development Processes (CMS, PCPI, and NCQA)

Figure A-1. CMS's Measure Development Process

Source: CRS Adaptation from CMS's "Quality Measure Development Overview." Accessed at http://www.cms.hhs.gov/QualityInitiativesGenInfo/downloads/QualityMeasuresDevelopmentOverview.pdf.

Figure A-2. PCPI's Measure Development Process

Source: CRS Adaptation from PCPI's "Measure Development Process."

Figure A-3. NCQA's Measure Development Process

Source: CRS Adaptation from NCQA's "HEDIS Measure Development Process." Accessed at http://www.ncqa.org/Portals/0/HEDISQM/Measure_Development.pdf.

Appendix B. NQF's Measure Endorsement Process

Figure B-1. The Steps in the NQF Measure Endorsement Process

Source: CRS Adaptation from the National Quality Forum, "The NQF Endorsement Process." Accessed at http://www.qualityforum.org/Publications/2008/10/The_NQF_Endorsement_Process.aspx.

Appendix C. List of Acronyms

AAFP: American Academy of Family Physicians

ACP: American College of Physicians

AHIP: America's Health Insurance Plans

AHRQ: Agency for Healthcare Research and Quality

AMA: American Medical Association

AQA: The AQA Alliance

CMS: Centers for Medicare and Medicaid Services

HEDIS: Healthcare Effectiveness Data and Information Set

HQA: Hospital Quality Alliance

IOM: Institute of Medicine

OMB: Office of Management and Budget

NCQA: National Committee for Quality Assurance

NPP: National Priorities Partnership

NQF: National Quality Forum

PCPI: Physician Consortium for Performance Improvement

PQRI: Physician Quality Reporting Initiative

RHQDAPU: Reporting Hospital Quality Data for Annual Payment Update

Footnotes

1.	Institute of Medicine (2001) "Crossing the Quality Chasm: A New Health System for the 21st Century," National Academy Press; Washington, DC; p. 1.
2.	Institute of Medicine (1999) "To Err is Human: Building a Better Health System." National Academy Press; Washington, D.C.
3.	Nolte, E., & McKee, C. M. (2008). "Measuring the health of nations: Updating an earlier analysis." Health Affairs, 27(1), 58-71. The conditions that are considered to be amenable to health care treatment include, but are not limited to, bacterial infections, treatable cancers, diabetes, cardiovascular and cerebrovascular disease, complications of common surgical procedures, and ischemic heart disease. The rate presented is an age-standardized death rate (SDR) per 100,000 population that is a simple average of the sex-specific SDRs, weighted for the distribution of males and females in the population.
4.	Midwest Business Group on Health. "Reducing the Costs of Poor-Quality Health Care Through Responsible Purchasing Leadership." April, 2003, pp. i-ii. http://www.mbgh.org/templates/UserFiles/Files/COPQ/copq%202nd%20printing.pdf.
5.	Schuster, M., McGlynn, E., & Brook, R. (1998). How good is the quality of health care in the United States? Milbank Quarterly, 76, 517-563.
6.	Mangione-Smith, R., Decristofaro, A. H., Setodji, C. M., Keesey, J., Klein, D. J., Adams, J. L., et al. (2007). The quality of ambulatory care delivered to children in the United States. The New England Journal of Medicine, 26(5), 644-649. E. A. McGlynn et al., "The Quality of Health Care Delivered to Adults in the United States", New England Journal of Medicine, 348(26), 2003
7.	Balas, E., Boren, A., Boren, S. (2000), "Managing clinical knowledge for health care improvement", Yearbook of Medical Informatics, pp. 65-70. National Library of Medicine, Bethesda, MD.
8.	Institute of Medicine. (2001) "Crossing the Quality Chasm: A New Health System for the 21st Century." National Academy Press; Washington D.C.; p. 232.
9.	A description of all of these activities is beyond the scope of this report. However, examples of quality research would include comparative effectiveness research, the development of clinical practice guidelines, or evidence syntheses. An example of quality measure development would include the development of the National Committee for Quality Assurance's Healthcare Effectiveness Data and Information Set (HEDIS) measures, and an example of quality reporting includes the Physician Quality Reporting Initiative (PQRI). Finally, examples of patient safety activity include the national level patient safety reporting system and nonpayment for "never events" policies.
10.	Donabedian, A. 1966. Evaluating the Quality of Medical Care. Milbank Memorial Fund Quarterly: Health and Society 44(3; pt. 2):166–203.
11.	The Agency for Healthcare Research and Quality (AHRQ). "Understanding Quality Measurement." Accessed at http://www.ahrq.gov/chtoolbx/understn.htm#domains.
12.	There are different types of composite measures, including appropriateness and opportunities measures. Appropriateness measures summarize performance across a group of measures related to a specific condition and require access to patient-level data. Opportunities measures are a basic summary across measures of opportunities for appropriate care, and are not reliant on patient-specific data, but rather may be calculated based on encounter data. AHRQ uses a number of composite measures in its National Healthcare Quality Report.
13.	Agency for Healthcare Research and Quality. "National Healthcare Quality Report 2008," at page 10. Accessed at http://www.ahrq.gov/qual/nhqr08/nhqr08.pdf on May 6, 2009.
14.	The National Quality Forum, a non-profit membership-based organization, aims to improve health care quality by setting national performance improvement priorities and endorsing quality measures.
15.	Section 1890(b)(1) of the Social Security Act.
16.	National Quality Forum. "Improving Healthcare Performance: Setting Priorities and Enhancing Measurement Capacity." Accessed at http://www.qualityforum.org/pdf/FINAL_2009NQFReporttoCongress.pdf on May 15, 2009.
17.	National Quality Forum. "NQF Time-Limited Endorsement." Accessed at http://www.qualityforum.org/pdf/txtime%20limited%20endorsement_csac_edits11-12-07.pdf on April 30, 2009.
18.	Accessed at http://www.jointcommission.org/NR/rdonlyres/D4B67088-980B-4642-A14F-ADB6805C1335/0/mktg_ahc.pdf on May 5, 2009.
19.	Riehle AI et al. (2007) "Specifying and Standardizing Performance Measures for Use at a National Level: Implications for Nursing-sensitive Care Performance Measures." Medical Care Research and Review 64(2):64S-81S at page 73S.
20.	NQF's measure evaluation criteria include whether a measure is linked to gains in quality and health outcomes; whether a measure is reliable and valid; whether a measure is usable (results are easy to understand and useful); and whether a measure is feasible (ease of access to required data). NCQA's desirable measure attributes include relevance; scientific soundness; and feasibility.
21.	Kmetik, K. (2007) "PCPI: What you should know about Consortium performance measures."Current Clinical Practice 1(1).
22.	CMS. "Quality Measures Development Overview." Accessed at http://www.cms.hhs.gov/QualityInitiativesGenInfo/downloads/QualityMeasuresDevelopmentOverview.pdf on May 14, 2009.
23.	CMS. "Roadmap for Quality Measurement in the Traditional Medicare Fee-for-Service Program." Accessed at http://www.cms.hhs.gov/QualityInitiativesGenInfo/downloads/QualityMeasuresDevelopmentOverview.pdf on May 14, 2009.
24.	PHS Act § 902(a); 42 U.S.C. § 299a.
25.	The Agency for Healthcare Research and Quality (AHRQ) maintains a public database of all health care quality measures, the National Quality Measures Clearinghouse (NQMC). Of the 1,762 measures currently in the database, 356 are NQF-endorsed. Accessed at http://www.qualitymeasures.ahrq.gov/ on April 28, 2009.
26.	Section 1848(k)(2)(B)(i) of the Social Security Act.
27.	Section 1848(k)(2)(C)(i) of the Social Security Act. The contracting entity referred to here is the National Quality Forum. There is an exception provided in statute for cases where feasible and practical endorsed measures are not available in a specific clinical area. In these cases, the Secretary may specify a measure that is not endorsed. See section 1848(k)(2)(C)(ii).
28.	Section 1886(b)(3)(B)(viii)(V) of the Social Security Act.
29.	Office of Management and Budget. Circular A-119. Accessed at http://www.whitehouse.gov/omb/circulars/a119/a119.html on April 30, 2009.
30.	15 U.S.C. 272 note.
31.	National Quality Forum. "The NQF Endorsement Process." Accessed at http://www.qualityforum.org/pdf/factsheet_endorsement%20100908_FINAL.pdf on April 30, 2009.
32.	73 Federal Register 69834 (November 19, 2008).
33.	73 Federal Register 38566 (July 7, 2008).
34.	Riehle AI et al. (2007) "Specifying and Standardizing Performance Measures for Use at a National Level: Implications for Nursing-sensitive Care Performance Measures." Medical Care Research and Review 64(2):64S-81S at page 67S.
35.	Pilot testing of EHR-based reporting is currently underway for 10 of the 2009 measures, but incentive payments will not be based on measures submitted in this manner.
36.	CMS describes measure groups as being a group of at least four clinically-related measures that share a common denominator. The groups include diabetes mellitus, preventive care, chronic kidney disease, coronary artery bypass graft surgery, rheumatoid arthritis, back pain, and perioperative care. http://www.cms.hhs.gov/PQRI/15_MeasuresCodes.asp#TopOfPage, accessed May 20, 2009.
37.	73 Federal Register 38566-7 (July 7, 2008).
38.	Section 1886(b)(3)(B)(viii)(IV) of the Social Security Act.
39.	This stage of expansion was statutorily mandated by Section 1886(b)(3)(B)(viii)(III) of the Social Security Act.
40.	CMS-1406-P, May 4, 2009, at page 335. Accessed at http://federalregister.gov/OFRUpload/OFRData/2009-10458_PI.pdf on May 20, 2009.
41.	Ibid.
42.	Werner RM and ET Bradlow. (2006) "Relationship Between Medicare's Hospital Compare Performance Measures and Mortality Rates." JAMA 296(22): 2694-2702.