Congress and the executive, as well as outside organizations, have long been attentive to the evaluation of federal programs, with frequent interest paid to the independent status of the evaluator. This interest continues into the current era, with numerous illustrations of the multi-faceted approaches adopted and proposed.
An evaluation may provide information at any stage of the policy process about how a federal government policy, program, activity, or agency is working. Congress has required evaluations through legislation (or requested these via its committee and Member offices); and the executive branch has pursued evaluations through presidential or agency directives.
Part of choosing how to carry out an evaluation involves deciding if some kind of “independence” would be a desirable attribute. Observers often see independence as a means of avoiding or deterring bias and ensuring an objective, impartial assessment. In the context of evaluation, independence may apply to an evaluation or to an evaluator. On one hand, for example, the term may relate to independence of an evaluation from the policy preferences of an individual or group (“independent evaluation”), perhaps by prohibiting political appointees from revising or evaluating a program. Independence may refer to an entity that conducts evaluations that also is located outside the immediate organization responsible for policy implementation (“independent evaluator”).
There is some diversity of opinion regarding the definition of independence and how it might be ensured. For example, an evaluator’s “external” status, outside the organization that is implementing a program, does not necessarily equate with independence. Nor would an evaluator’s “internal” status, inside the implementing organization, necessarily equate with a lack of independence for an evaluation (e.g., if an expert panel reviewed the internally produced evaluation for bias). There is varying opinion concerning when independence is necessary, or possibly counterproductive, and what value it may bring.
The differences of opinion among definitions and perceived need notwithstanding, instances of independent program evaluators appear to be growing in number and variety at the federal level.
This report focuses on examples of independent evaluators (IEs): when an evaluation is to be conducted by an entity outside the immediate organization that is responsible for policy implementation, and the entity also is intended to have one or more dimensions of independence. IEs and similar constructs, however, vary across a number of characteristics and attributes: structure, jurisdiction, authority, resources, length of tenure, and specific duties and responsibilities. These differences, in turn, could affect their capabilities, effectiveness, and assistance to others, including their contributions to the oversight of a program or project by Congress and the executive branch.
After an overview of such entities—which encompass new units created specifically for conducting an evaluation as well as existing ones, such as the Government Accountability Office and offices of the inspectors general—this report suggests possible broad characteristics and criteria of independent evaluators or similar units, which could be valuable in oversight or legislative endeavors. The Appendix describes a number of such offices—past, present, and proposed—along with citations to relevant official documents and other materials for each example (public laws, legislative proposals, executive branch directives, and secondary analyses). This report will be updated as conditions warrant.
Congress and the executive, as well as outside organizations, have long been attentive to the evaluation of federal programs, with frequent interest paid to the independent status of the evaluator. This interest continues into the current era, with numerous illustrations of the multi-faceted approaches adopted and proposed.
An evaluation may provide information at any stage of the policy process about how a federal government policy, program, activity, or agency is working. Congress has required evaluations through legislation (or requested these via its committee and Member offices); and the executive branch has pursued evaluations through presidential or agency directives.
Part of choosing how to carry out an evaluation involves deciding if some kind of "independence" would be a desirable attribute. Observers often see independence as a means of avoiding or deterring bias and ensuring an objective, impartial assessment. In the context of evaluation, independence may apply to an evaluation or to an evaluator. On one hand, for example, the term may relate to independence of an evaluation from the policy preferences of an individual or group ("independent evaluation"), perhaps by prohibiting political appointees from revising or evaluating a program. Independence may refer to an entity that conducts evaluations that also is located outside the immediate organization responsible for policy implementation ("independent evaluator").
There is some diversity of opinion regarding the definition of independence and how it might be ensured. For example, an evaluator's "external" status, outside the organization that is implementing a program, does not necessarily equate with independence. Nor would an evaluator's "internal" status, inside the implementing organization, necessarily equate with a lack of independence for an evaluation (e.g., if an expert panel reviewed the internally produced evaluation for bias). There is varying opinion concerning when independence is necessary, or possibly counterproductive, and what value it may bring.
The differences of opinion among definitions and perceived need notwithstanding, instances of independent program evaluators appear to be growing in number and variety at the federal level.
This report focuses on examples of independent evaluators (IEs): when an evaluation is to be conducted by an entity outside the immediate organization that is responsible for policy implementation, and the entity also is intended to have one or more dimensions of independence. IEs and similar constructs, however, vary across a number of characteristics and attributes: structure, jurisdiction, authority, resources, length of tenure, and specific duties and responsibilities. These differences, in turn, could affect their capabilities, effectiveness, and assistance to others, including their contributions to the oversight of a program or project by Congress and the executive branch.
After an overview of such entities—which encompass new units created specifically for conducting an evaluation as well as existing ones, such as the Government Accountability Office and offices of the inspectors general—this report suggests possible broad characteristics and criteria of independent evaluators or similar units, which could be valuable in oversight or legislative endeavors. The Appendix describes a number of such offices—past, present, and proposed—along with citations to relevant official documents and other materials for each example (public laws, legislative proposals, executive branch directives, and secondary analyses). This report will be updated as conditions warrant.
An evaluation may provide information at any stage of the policy process about how a federal government policy, program, activity, or agency is working. The resulting information may be used to better understand policy problems and to help inform the design, implementation, and oversight of policies. A variety of stakeholders may seek evaluation information and data to help them exercise their responsibilities, obligations, and rights in a representative democracy, including the establishment, implementation, oversight, and other requirements of a federal program or project. These stakeholders and interested parties include, among others, Congress, the President, federal agency officers and employees, state and local governments, interest groups, private sector contractors, the news media, and the general public.
For a number of reasons, there is not necessarily a best way to design and carry out an evaluation of a particular program, activity, or operation. A program's context, including its longevity and the manner in which it is implemented, typically influences choices about evaluation design. In addition, stakeholders often have different needs and research questions about which they would like more information.1 One aspect of choosing how to carry out an evaluation involves deciding when some kind of "independence" would be a desirable attribute.
In the context of evaluation, independence may apply to an evaluation or to an evaluator.2 On one hand, for example, the term may relate to independence of an evaluation from the policy preferences of an individual or group ("independent evaluation"), perhaps by prohibiting political appointees from revising an evaluation. Independence also may refer to an entity that conducts evaluations that also is located outside the immediate organization responsible for policy implementation ("independent evaluator").
There is some diversity of opinion regarding how to define independence and what makes an evaluator (or evaluation) independent. For example, an evaluator's "external" status, outside the organization that is implementing a program, does not necessarily equate with independence. Nor would an evaluator's "internal" status, inside the implementing organization, necessarily equate with a lack of independence for an evaluation (e.g., if an expert panel reviewed the internally produced evaluation for bias). There is also varying opinion concerning when independence is necessary, or possibly counterproductive, and what value it may bring. In some situations, there may be little need for independence in evaluations (e.g., when equipping a program with capacity to improve its own operations iteratively, or when "adaptively" managing a project to help restore an ecosystem).3 Nonetheless, instances of independent evaluators appear to be growing in number and variety.
This report focuses on the characteristics of independent evaluators (IEs)—examples of which are described in the Appendix—when an evaluation is to be conducted by an entity outside the immediate organization that is responsible for policy implementation, and the IE is also intended to have one or more dimensions of independence. IEs and similar constructs that conduct independent evaluations, however, vary across a number of attributes: structure, jurisdiction, authority, resources, length of tenure, and specific duties and responsibilities. These differences, in turn, could affect their capabilities, effectiveness, and assistance to others, including their contributions to the oversight of a program or project by the executive or legislature.
After an overview of such entities—which include newly and specially created units as well as existing ones, such as the Government Accountability Office (GAO) and offices of inspector general (OIGs)—this report suggests possible broad characteristics and criteria of independent evaluators or similar units. In situations when some extent of independence in an existing or proposed evaluator were deemed desirable, Congress might consider the different kinds and degrees of independence that could be pursued. The final section describes a number of such offices, along with citations to relevant materials for each example (public laws, legislative proposals, executive branch documents, and secondary analyses).
Evaluation's roots have been characterized as reaching as far back as hundreds of years or, even more remarkably, thousands of years.4 This lengthy heritage notwithstanding, some observers describe the "modern era" of evaluation, and particularly "program evaluation" (focusing on government social programs), as emerging in the 1960s and growing appreciably since then.5 Expansion of social, economic, and environmental programs, among others, in many cases has been accompanied by legislation mandating and funding evaluation or by executive directives ordering such studies. These changes facilitated and increased the diversification of program evaluation, based on a perceived need to better inform the understanding of policy problems, formulate responses, and strengthen oversight by the executive and legislature over a growing expanse and complexity of federal programs. Accompanying this development, program evaluation has also become more sophisticated and selective in terms of its methods, methodology, type and level of operation, and subject or policy area.6
The 1990s gave added impetus to assessments, evaluations, and reviews of programs, activities, and operations through a number of laws, which remain on the books. Among these supporting evaluation efforts, directly or indirectly, are the Chief Financial Officers Act of 1990 (CFO Act; P.L. 101-576, 104 Stat. 2838); Government Performance and Results Act of 1993 (GPRA; P.L. 103-62, 107 Stat. 285); Information Technology Management Reform Act of 1996, later renamed the Clinger-Cohen Act (CCA; Division E of P.L. 104-106, 110 Stat. 679, and P.L. 104-208, 110 Stat. 3009-393); and Federal Financial Assistance Improvement Act of 1999 (P.L. 106-107, 113 Stat. 1486).7 These and other earlier relevant statutes, including the Inspector General Act of 1978 (IG Act), as amended,8 allow for flexibility in determining various evaluation requirements, particularly concerning individual projects and programs. GPRA, for example, defines "program evaluation" quite generally as "an assessment, through objective measurement and systematic analysis, of the manner and extent to which Federal programs achieve intended objectives."9
As noted earlier, independent evaluation oftentimes refers to the review and assessment of how well programs and projects are working that is conducted by a unit outside the program office itself. Such units have been expressly authorized to perform independent evaluations or have implied authority to do so under a broad mandate to oversee or review a program, project, activity, or operation.
Independent evaluation can be, and has been, carried out by positions under a number of different names and titles. Specific ones identified in this study include independent evaluator, independent auditor, interagency coordinating research institute, accountability board, peer reviewer, independent or peer review panel, program evaluator, program evaluation unit, research and evaluation institute, inspection and evaluation (I&E) unit in an office of inspector general (OIG),10 and various organizational groupings in the Government Accountability Office.11 These have been established within the parent agencies (but separate from the program office) or in outside organizations, public or private. In addition, such entities, whatever their name and wherever located, appear to be growing in number and in a variety of policy domains.
The concepts of independent evaluator and independent evaluation, however, have led to generalized understandings, rather than a precise, detailed, agreed-upon definition. The broad notions lack specification, standardization, and uniformity. There appears to be no express, across-the-board definition of an "independent evaluation" or "independent evaluator" in federal statute or regulation.12 (Further discussion of these concepts appears below.) As a result, there are significant variations among offices which may fall under the same rubric or which conduct the same function. In other words, "independent," "evaluator," and "evaluation" are subject to different meanings among offices that use similar terminology.
The research for this analysis—which identifies a number of independent evaluators or similar entities and functions—found that none of the entities were identical. Although there were similarities governing several key characteristics in some cases, there were substantial differences of structure, organization, authority, jurisdiction, funding, staffing, length of tenure, or duties and responsibilities.
Dissimilarities among these characteristics in IE-like positions and functions, in fact, appear to be more common than their similarities. This arises, in part, because of different rationales, expectations, research questions, and conditions surrounding the establishment of the positions. Because of this, the resulting independent evaluators lack standardization and uniformity across-the-board, and a number have been given substantial flexibility in organizing their own operation. Structural differences might also arise, at least in some cases, because the creator of the entity (the President, agency head, or Congress) intends to allow for discretion and flexibility at the implementing level (for the agency or IE, reflecting his or her expertise and experience). Along with this, the establishing authority for many positions—public law, executive order, or administrative fiat—often do not specify certain characteristics of the office, such as its funding, reporting requirements, tenure, or evaluation standards.
Disagreements, moreover, have arisen over the value, importance, and means of supporting independence for evaluators. The Government Accountability Office (GAO), when it examined the Office of Management and Budget's (OMB's) guidance for the Program Assessment Rating Tool (PART) in 2005, found that a
source of tension between OMB and agency evaluation interests was the evaluation's independence. PART guidance stressed that for evaluations to be independent, nonbiased parties with no conflict of interest, for example, GAO or an Inspector General should conduct them .... OMB subsequently revised the guidance to allow evaluations to be considered independent if the program contracted them out to a third party or they were carried out by an agency's program evaluation office. However, disagreements continued on the value and importance of this criterion.13
Furthermore, an evaluation scholar noted that independence may not be an important dimension of an evaluation if the objectives of the evaluation primarily focus on strengthening institutions or building "agency or organizational capacity in some evaluative area."14
Other considerations tie into program evaluation in general, no matter where it is carried out. These could extend to whether the subject of the evaluation is either (1) too large and broad an undertaking for the evaluator, thereby overwhelming him or her; or (2) to the contrary, too small and narrowly focused to be of utility apart from the immediate project (in developing, for instance, recommendations for best practices or the entire program). Along the same lines are concerns about whether the evaluation is integrated into the overall operation of a program and whether an evaluator might be perceived as "usurping" or competing with the prerogatives of a parallel office. Others, discussed further below, deal with selecting the most appropriate evaluation technique or method, especially in light of the diversity among them, as well as the right evaluator.
Independent evaluation entities do not arise or exist in a vacuum. Their establishment, powers, performance, effectiveness, and impact are subject to a variety of influences. And because they are usually ad hoc and idiosyncratic, they differ in their structure and operation. IE-like offices range from modest efforts—beginning with a position without separate funding or staffing, evaluating only a single, short-term project—to major undertakings, extending to a separate unit composed of a number of individuals, with its own budget and resources, operating continuously throughout the life of a long-term, interagency, and sometimes intergovernmental program. In between these ends of the spectrum are a number of possible combinations. Variations exist even among units evaluating programs in the same broad subject area. Six of the examples below, for instance, deal with interagency, intergovernmental, and, in some cases, interstate waterway programs; yet the evaluation units differ from one another, sometimes significantly, in their main characteristics or in the specifications in their establishing authority.
In sum, the characteristics of independent evaluators and similar constructs are determined by a number of formal and informal factors. Differences among evaluation constructs are reflected in the powers and protections in the authorities that established the IEs—public laws, executive orders, or administrative directives—as well as in their relationships with executive officials. Besides the variables already identified, other influences, some informal or intangible, add to this mixture. These include the expertise of the staff, competency and impartiality of the evaluator, and trust and confidence between an evaluator and the program office. These, in turn, affect the IEs' actual and potential independence, capacity, capability, and effectiveness—including their contributions to oversight by Congress and the executive—extending from the immediate program office to the parent agency, OMB, and the President.
As emphasized above, the terms "independent evaluator" and "independent evaluation" lack precise, standardized, agreed-upon definitions in public law or executive directive, and they vary considerably across a number of dimensions. Nonetheless, many possible criteria and attributes of an independent evaluator or similar construct can be identified, based on the research for this examination and on other sources, both public and private. These sources include the American Evaluation Association (AEA);15 evaluation reference works, such as the Encyclopedia of Evaluation;16 the Joint Committee on Standards for Educational Evaluation (Joint Committee);17 and federal agencies that conduct and direct evaluations of programs, projects, activities, and operations. Illustrative agencies include the Government Accountability Office (GAO),18 the Council of the Inspectors General on Integrity and Efficiency (CIGIE) and its predecessors,19 and the Office of Management and Budget (OMB).20
These different sources cover common ground, including the concepts of evaluation and independence, ways in which the terms may be understood, and how the concepts may be specified. Differences exist, especially in how detailed, specific, and elaborate these descriptions are, ranging from a short statement on a concept to a lengthy listing and interpretation of relevant standards.
AEA, which promotes evaluation for public programs, adopts a broad understanding of the concept of evaluation, which can occur throughout the life of a program: "Evaluation is a field that applies systematic inquiry to help improve programs, products, and personnel, as well as the human actions associated with them."21 While recognizing that evaluators' work can vary greatly, AEA holds that
the common ground for all evaluators is that they aspire to achieve accountability and learning by providing the best possible information that might bear on the value of whatever is being evaluated.... Evaluations prepared by professional, independent evaluators help prevent information gaps by: improving knowledge and understanding of how programs work, strengthening public accountability, assessing program effectiveness and efficiency, and identifying opportunities and pathways to achieving objectives, outcomes, and efficiencies.22
To support these goals, AEA specifies "several of the key elements of a national framework for evaluation practices."23 These benchmarks include
In July 2004, AEA approved a set of guiding principles for evaluators, which interrelate with one another as well as with the foregoing standards.25 The five, each of which is detailed in its brochure, are:
The Encyclopedia of Evaluation offers a working definition of "independent evaluation":
For an evaluation to be considered independent, the evaluator must be impartial, objective, unencumbered, and balanced. Further, because perceived independence is as important as independence itself, the evaluator must be accountable for every step in the research process and able to document all key decisions and actions for the client organization, or other evaluators, and the community at large. Overall, external evaluations tend to hold more credibility than internal ones because the external evaluator appears to have less to gain or lose from the evaluation findings and is less likely to experience a conflict of interest.27
Notably, this definition does not appear to equate automatically an evaluator's "external" status with independence, or an evaluator's "internal" status with lack of independence.
The Joint Committee's standards for evaluating educational programs might be adapted to other fields.28 It posits that "sound evaluations of educational programs, projects, and materials in a variety of settings should have four basic attributes."29 The associated standards govern:
GAO explains the concept of program evaluation by including within its scope evaluations conducted by external entities, experts inside the agency that contains a program, and the employees responsible for implementing programs and policies:
Program evaluations are individual systematic studies conducted periodically or on an ad hoc basis to assess how well a program is working. They are often conducted by experts external to the program, either inside or outside the agency, as well as by program managers. A program evaluation typically examines achievement of program objectives in the context of other aspects of program performance or in the context in which it occurs. Four main types can be identified, all of which use measures of program performance, along with other information, to learn the benefits of a program or how to improve it.31
The 2008 Inspector General Reform Act has directed the new Council of the Inspectors General on Integrity and Efficiency to "develop plans for coordinated, Governmentwide activities that address these problems and promote economy and efficiency in Federal programs and operations, including interagency and interentity audit, investigation, inspection, and evaluation programs and projects to deal efficiently and effectively with those problems concerning fraud and waste that exceed the capability or jurisdiction of an individual agency or entity."32 This call to action to all IGs builds on a growing movement among individual offices to increase and enhance program evaluation.
The IG community generally has considered evaluation as part of a more encompassing function of inspection: "An inspection is defined as a process that evaluates, reviews, studies, and/or analyzes the programs and activities of a Department/Agency" for a number of purposes.33 The CIGIE Inspection and Evaluation Committee, which drafted the standards, included among these purposes providing factual information to managers for decisionmaking; monitoring compliance; measuring performance; assessing the efficiency and effectiveness of programs and operations; sharing best practices; identifying where administrative action may be necessary; and making recommendations for improvements to programs, policies, or procedures.34 To accomplish this, the I&E Committee developed standards dealing with 14 separate matters: competency; independence; professional judgment; quality control; planning; data collection and analysis; evidence; records maintenance; timeliness; fraud, other illegal acts, and abuses; reporting; follow-up; performance measurement; and working relationships and communications.35
The Inspection and Evaluation Committee, however, qualifies the adoption of the standards: "While these standards are advisory, and compliance is voluntary, their consistent application is encouraged."36 The panel adds that "the inspection function at each Department/Agency is tailored to the unique mission of the respective Department/Agency."37 Although the inspection function did not originate when the IG Act was approved in 1978, it has grown over time in terms of complexity and diversity as well as size; that is, the number of OIG units, budget, and staff dedicated to it.38 The result is that most—but not all—IG offices conduct inspections; and these I&E units vary in expertise, location, type of work, longevity, staffing, and budget.39
Over the past decade, OMB has also offered guidance on how to achieve independence and expertise in evaluations. Its directions and instructions, however, exhibit some differences in specifications and orientation between the Bush and Obama Administrations.
Considerations about the independence and orientation of program evaluation were included in the operation of the Program Assessment Rating Tool (PART), which was developed during the Administration of George W. Bush but is no longer in effect.40 Although the guidance did not define independent evaluation or evaluator expressly, OMB referred to its role in several instances. One instance, about the scope and quality of "independent evaluations," stated,
Purpose: To ensure that the program or agency conducts non-biased evaluations on a regular or as-needed basis to fill gaps in performance information. These evaluations should be of sufficient scope and quality to improve planning with respect to the effectiveness of the program.41
OMB added that such independent evaluations were to be of "high quality, sufficient scope, unbiased, independent, and conducted on a regular basis to support program improvements."42
As noted previously in this report, however, GAO discussed how disagreements arose in this context over how to define independence for an evaluator and whether independence was necessary or of value in some circumstances.43 OMB initially advocated for a separate, outside entity—in particular, statutory IGs or GAO—to conduct the evaluation. Agencies balked at this, insisting that experienced program evaluation offices within an agency, for instance, could conduct such evaluations with independence. OMB reconsidered its stand and allowed that "evaluations could be considered independent if the program contracted them out to a third party or they were carried out by an agency's program evaluation office."44 OMB also recognized a possible conflict between competing analyses: "The program should defend differences if an independent entity's analysis differs from the program's analysis."45
After Barack Obama became President, the Office of Management and Budget issued several memoranda and a guidance dealing with independent program evaluation. OMB Director Peter R. Orszag issued two memoranda on the subject—one in 2009 and another the next year—which began with a nearly identical premise: "Rigorous, independent program evaluations can be a key resource in determining whether government programs are achieving their intended outcomes as well [effectively] as possible and at the lowest possible cost."46
By comparison to the PART pronouncements, the Obama OMB assumed a somewhat different stance on how to define independence. In the October 2009 memorandum, OMB concentrated more on the concept of independent evaluation instead of an independent evaluator.47 Notably, this memorandum made repeated reference to conducting "rigorous, independent studies that are free from political interference." The memorandum did not provide more detail on what constitutes political interference or what this sort of independence might look like. When an agency applied for Administration support in the President's budget for capacity-building funds, however, OMB required agencies to address explicitly their ability to conduct studies that are free from political interference.
In its follow-up memorandum, OMB added to this in its "evaluation initiative for FY 2012," allocating, in the President's upcoming budget proposal, "approximately $100 million to support 35 rigorous program evaluations and evaluation capacity-building proposals across the Federal government."48 In so doing, the evaluation initiative repeated the earlier call for an "assessment of agency capacity to conduct rigorous, independent evaluations ... that are free from political interference."49 A complementary goal—which also suggested several ways to institutionalize independent evaluation within an agency—is to
attract and retain talented researchers in an office with standing within the agency. Agencies are encouraged to propose changes or reforms that are needed to meet these objectives and may request funds to strengthen their internal evaluation expertise and processes. Several existing models may be worthy of consideration, including a congressionally chartered institute within an agency, or an office headed by a senior official reporting directly to the Secretary or Deputy Secretary.50
The 2009 memorandum had also announced an "increased emphasis" on program evaluations and, more specifically, "impact evaluations."51 The 2010 memorandum followed suit, recognizing the need to overcome the absence of completed or timely program evaluations for many federal programs. As a result, the Office called on agencies to include funding for FY2011 to conduct evaluations or build capacity, to correct these defects and strengthen program evaluation.52 Extending this orientation, another OMB memorandum emphasized a change in the objective of evaluations and their measurements. This OMB memorandum, also issued in mid-2010, states that
the Administration is transitioning from a planning and reporting approach focused primarily on the supply of performance information to .... performance improvement strategies .... [with] Unrelenting attention to achieve the ambitious, near-term performance goals that agency leaders identify as High Priority Performance Goals ("Priority Goals") in the President's FY 2011 Budget; Establishing constructive performance review processes in agencies that are sustained over time; and Making Government Performance and Results Act documents useful.53
A 2010 report by the Government Accountability Office, which was released before OMB's initiatives in the same year, examined attempts by both the Bush and Obama Administrations to improve government efficiency, including ways it connects to PART and its successor.54 GAO found that although "most programs developed an efficiency measure," most of these failed to adopt both input and output or outcome measures; and agency officials described "challenges to developing and using program-level efficiency measures and performance measures in general."55 Nonetheless, "officials for some programs stated that the efficiency measures reported for PART were useful" in a variety of ways, including "to evaluate proposals from field units."56 The GAO report also recognized improvements coming from both the Bush and Obama Administrations along with the lengthy history of legislative reform efforts that tie efficiency into program effectiveness, particularly in the Government Performance and Results Act.57
The two subsequent 2010 OMB memoranda recognized some of the same goals. Its memorandum on Evaluating Programs for Efficacy and Cost-Efficiency (2010), as the title suggests, dealt with the relationship between efficacy and efficiency and the need to measure both. And OMB's Performance Improvement Guidance added to this, specifying that evaluation should focus on "performance improvement strategies" in general and on "making GPRA documents useful" in particular.58
The aforementioned understandings of terms like "evaluation" and "independence" are not hard-and-fast rules. Rather, they might be considered as perspectives, some of which are similar. Using the perspectives and some of the examples cited later in this report, a number of possible criteria for an independent evaluator can be identified. These characteristics, which are not exhaustive, are intended only to identify a range of possibilities and options to consider for establishing an independent evaluator, modifying evaluative operations and organizations, or reviewing an evaluator's methodology and findings. CRS takes no position on the advisability of independence or the adoption of any particular characteristic.
Based on the foregoing analysis and the examples cited later in this report, the following criteria illustrate a range of factors that could affect the independence of an IE-like entity, which, in turn, could help to determine the IE's capacity, capability, credibility, impact, and effectiveness.
These criteria also suggest a range of considerations in establishing a new IE entity, or reviewing and modifying an existing one. A number of choices about organizational design and operations might be considered that could support the independence, quality, reliability, and meaningfulness of an IE's studies and reports to a variety of stakeholders, including Congress, the President, agency employees, state and local governments, interest groups, private contractors, and the public. These outcomes are likely to be enhanced by clear and specific requirements regarding the purposes, responsibilities, and duties of the IE entities as well as protections for their officers. Posed as questions, these considerations, which closely relate to one another and in some circumstances may overlap, include the following:
In the understanding adopted here, truly independent evaluators are not found within the program office itself, because that location would (appear to) compromise their independence. Even with this qualification, however, independent evaluators of federal programs appear to have increased since the 1960s and continue to do so. IEs exist both inside the agency proper—in auditing offices, for instance, or, as a "model" raised by OMB, in a "congressionally chartered institute"—as well as outside it. Independent evaluators and comparable entities have been created as new, separate units, operating under a variety of names and titles. And evaluations have been undertaken by appropriate existing agencies, including the Government Accountability Office, offices of inspector general, or various other government constructs, such as NIST and the National Academy of Sciences.
IEs and similar constructs, moreover, are multi-faceted and diverse, differing in their establishing authority, structure, organization, responsibilities, and requirements, among other characteristics. Such differences have come about for a number of reasons, including the following:
Following are descriptions of a number of independent evaluation entities, as set up by their establishing authority or proposed. These were selected through a search of public laws, executive orders, other administrative directives, and legislative proposals that specifically call for an "independent evaluation," "independent evaluator," or similar devices, including "peer review" or "independent study." The listing is not comprehensive. For instance, it does not include relevant reports from the Government Accountability Office, because of their substantial number. And it does not include relevant reports from inspectors general, unless these are specifically called for in legislation. Nonetheless, the descriptions of the selections illustrate the diversity of independent evaluation units and their adoption over time, particularly in the contemporary era. (For each entry, the title of the entity uses the lower or upper case as it appears in the authorization or proposal and is in bold type for emphasis.)
Based on these examples, no single entity meets all the characteristics that could be addressed in its establishing authority, although some come close. The selections also demonstrate a wide range and diversity among entities which have been empowered to conduct evaluations. As a corollary, none of the evaluation offices are identical to any other in all respects. Differences are observable in the specifics associated with: appointment, tenure, jurisdiction, evaluation criteria and standards, consultation with outside organizations, reporting mandates, reports' recipients and responses, and funding and staffing, among other characteristics.
Chesapeake Bay Protection and Restoration Independent Evaluator
(Executive Order 13508) 61
The independent evaluator called for in Executive Order 13508, "Chesapeake Bay Protection and Restoration," issued by President Obama on May 12, 2009, is to assist a new Federal Leadership Committee for the Chesapeake Bay in its oversight of the program. The committee, established in the order, comprises representatives from a number of federal departments and agencies; it was established "to begin a new era of shared Federal leadership with respect to the protection and restoration of the Chesapeake Bay."62 The committee's responsibilities are "to oversee the development and coordination of programs and activities," along with managing "the development of strategies and program plans for the watershed and ecosystem of the Chesapeake Bay and oversee their implementation."63 Among other things, the committee is to prepare and publish a strategy, following submission of relevant draft reports from lead federal agencies. The overarching strategy is to include, among other things: defining environmental goals; identifying key measurable indicators or environmental condition and changes; describing specific programs and strategies to be implemented; identifying the mechanism to assure that governmental and other activities, including data collection and distribution, are coordinated; and "describing a process for the implementation of adaptive management principles, including a periodic evaluation of protection and restoration activities."64
These requirements, directly or indirectly, tie into the functions of an independent evaluator called for in the executive order. In order to "strengthen accountability" over the program,
[t]he Committee, in collaboration with State agencies, shall ensure that an independent evaluator periodically reports to the Committee on progress toward meeting the goals of this order. The Committee shall ensure that all program evaluation reports, including data on practice or system implementation and maintenance funded through agency programs, as appropriate, are made available to the public by posting on a website maintained by the Chair of the Committee.65
Key characteristics of the Chesapeake Bay Restoration independent evaluator position—based on this provision and portions of the Federal Leadership Committee's mandate—are:
Proposed Chesapeake Clean Water and Ecosystem Restoration Act of 2009 Inspector General and Chesapeake Bay Program Scientific and Technical Advisory Committee66
This bill (S. 1816, 111th Congress), which focuses on restoration of Chesapeake Bay, provides for the Inspector General of the Environmental Protection Agency to evaluate implementation of the enactment and for a special Scientific and Technical Advisory Committee. The legislation is intended to expedite and enhance the restoration of the Chesapeake water and ecosystem. It would reauthorize the Chesapeake Bay Program, expand state and local government authority, provide new grant authorizations, and strengthen enforcement tools. In so doing, it would augment existing legislation (the Federal Water Pollution Control Act); President Obama's Executive Order 13508 (described above); and the on-going Chesapeake Bay Program, which has been a federal, state, and local government charge since the mid-1980s. The Chesapeake Executive Council would direct the Program, in accordance with a Chesapeake Bay Agreement. The Council would develop and implement management strategies and plans for the Program. The operational Office of the Council would remain in EPA, providing support for it in a number of ways. These include implementing and coordinating science, research, monitoring, and data collection; developing and maintaining pertinent information; assisting the signatories of the Agreement in developing and implementing action plans; coordinating the activities of EPA and other entities in developing strategies to improve water quality and living resources of the Bay; and implementing outreach programs for public information, education, and participation to foster stewardship of the bay. Along with this, the EPA Administrator would be authorized to enter into agreements with other federal agencies to carry out these matters. If enacted, S. 1816 would provide for various grants to public and private entities, for technical and other assistance as well as implementation and monitoring.
In accordance with E.O. 13508, S. 1816 would also require the EPA Administrator to issue an annual action plan—describing how federal funding in the President's budget submission to Congress would be used to protect and restore the Chesapeake Bay—and biennial progress reports assessing relevant developments and indicators.
The bill also provides that the "Inspector General of the Environmental Protection Agency shall evaluate the implementation of this section on a periodic basis of not less than once every three years." No other particulars about such an evaluation are specified in the proposal. The legislation would also require the EPA Administrator to consult with several different entities about particular matters. One of these directives is "to consult with the Chesapeake Bay Program Scientific and Technical Advisory Committee regarding independent review of [certain] monitoring designs."
Proposed Chesapeake Bay Accountability and Recovery Act of 2009 Independent Evaluator67
This proposed legislation (H.R. 1053, 111th Congress) parallels some of the main provisions in E.O. 13508, both of which deal with the Chesapeake Bay restoration and include an Independent Evaluator. The House bill, as approved by the House Committee on Natural Resources, calls for an "interagency crosscut budget" for restoration activities in the Chesapeake Bay watershed, with funding, expenditures, and accounting requirements detailed (Sec. 1).68 This crosscut budget, composed by the various agencies involved in the restoration program, is to be submitted to Congress by the Director of the Office of Management and Budget, after consultation with the Chesapeake Bay Executive Council, the chief executive of each Chesapeake Bay State (the definition of which includes the District of Columbia), and the Chesapeake Bay Commission. In addition, H.R. 1053 (Sec. 3) requires that the Administrator of the Environmental Protection Agency (EPA), "in consultation with other Federal and State agencies, shall develop an adaptive management plan for the Chesapeake Bay Program and restoration activities that includes," among other matters: a definition of specific and measurable objectives to improve water quality, habitat, and fisheries; a process for stakeholder participation; monitoring, modeling, experimentation, and other research and evaluation practices; a process for modifying restoration priorities that have not attained or will not attain the aforementioned specific and measureable objectives; and a process for prioritizing restoration activities and programs to which adaptive management is to be applied. The EPA Administrator is also required to submit annual reports on the implementation of the adaptive management plan.
The proposal adds an Independent Evaluator, appointed by the Administrator from among nominees submitted by the Chesapeake Executive Council. The IE is to "review and report on restoration activities and the uses of adaptive management, including on such related topics as are suggested" by the Council (Sec. 4). The IE "shall submit a report to the Congress every 3 years in the findings and recommendations of reviews" with regard to the adaptive management plans and their implementation (Sec. 4). As noted above, the plans' inclusions are: developing measurable objectives; monitoring, modeling, experimentation, and other research and evaluation practices; and processes for modifying restoration activities as well as prioritizing restoration activities and programs.
Comprehensive Everglades Restoration Plan Independent Scientific Review Panel and Independent Peer Review69
The Independent Scientific Review Panel, established by a collective of interdepartmental and inter-governmental offices, is to be "convened by a body, such as the National Academy of Sciences, to review the plan's progress toward achieving the natural system restoration goals of the Plan."70 The panel is to produce a biennial report to Congress and members of the establishing body, that includes an assessment of the ecological indicators and other measures of progress in restoring the ecology of the natural system, based on the plan.
In addition, the enactment calls for an independent peer review of methods for project analysis.71 The statute charges the Secretary of the Army to contract with the National Academy of Sciences (with a specified authorization of appropriations), to conduct a study that includes reviews of various methods used and a comparative evaluation of the basis and validity of relevant state-of-the-art methods. The report—to be issued not later than one year after the date of the contract and submitted to the Secretary and to specified congressional committees—is to include the results of the study and specific recommendations for modifying any methods being used in the project for conducting economic and environmental analyses of water resource projects.
Prince William Sound (Alaska) Oil Spill Recovery Institute and Advisory Board72
The Oil Spill Recovery Institute, established by the Secretary of Commerce, is to conduct research and carry out educational and demonstration projects designed to develop the best available techniques, equipment, and materials dealing with oil spills. In addition, the institute is to complement federal and state damage assessment efforts as well as determine, document, assess, and understand the long-range effects of relevant oil spills. The institute is to publish and make available to any person the results of its research, educational, and demonstration projects; copies of all such materials are to be provided to the National Oceanic and Atmospheric Administration.
The policies of the institute are determined by an Advisory Board, which also appoints the institute's chair. The Advisory Board, itself chaired by the Secretary of Commerce, consists of federal and state officials, representatives of relevant industries, representatives of Alaska Natives, and residents of local communities. The board may also request a scientific review of a research program it authorized every five years, to be conducted by the National Academy of Sciences. The board shall establish a Scientific and Technical Committee, composed of relevant specialists, to advise and make recommendations to the board regarding the conduct and support of research, projects, and studies.
Oil Pollution Research Interagency Coordinating Committee73
The Oil Pollution Research Interagency Coordinating Committee includes representatives from a number of federal agencies. It is chaired by the representative of the United States Coast Guard and operates under a specified budget. The committee is to report to Congress on a plan dealing with oil pollution research, development, and demonstration program. At this initial stage, the enactment requires consultation with affected states on certain matters and contracting with the National Academy of Sciences for advice and guidance on preparing the plan and assessing its adequacy. Subsequently, the committee is to coordinate the establishment of a program for conducting oil pollution research and development, which includes requirements for monitoring and evaluating relevant aspects of the plan. The committee is also authorized to make recommendations for grants to the private sector to an appropriate granting agency represented on the committee; its recommendations are to ensure an appropriate balance within a region among various aspects of oil pollution research. The chairperson of the committee is to report to Congress every two years on its activities and on activities proposed and carried out under the act.
Water Resources Development Independent Peer Review Panels74
Various project studies authorized by the act are subject to peer review by independent panels of experts. Some reviews are mandatory, while others are discretionary; the latter include ones requested by a governor of an affected state or certain federal agencies. For each project subject to a peer review, the Chief of Engineers of the Army Corps of Engineers is to contract with the National Academy of Sciences or a similar independent scientific and technical advisory organization to establish a panel of experts, who are to represent a balance of areas of expertise suitable for the review and who have no conflict or direct involvement with the project being reviewed. Specifications for the peer review include assessing the adequacy and acceptability of economic, engineering, and environmental methods, models, and analyses.
Certain committees of Congress are to be notified of a project study for peer review prior to its initiation. In addition, the review panels are to submit a report to the Chief of Engineers not more than 60 days after the last day of public comment for a draft project study or later, if the chief determines an extension is necessary. The panels are also directed to submit to the chief a final report containing certain analyses. They may make recommendations, to which the chief shall issue a written response for any recommendation adopted or not adopted. Copies of the reports and the chief's responses are to be made available to the public by electronic means and transmitted to certain congressional committees. The enactment authorizes federal funds to be used for each panel; it imposes a ceiling of $500,000, which can be waived by the chief if he or she determines the waiver to be appropriate.
American Recovery and Reinvestment Act (ARRA) of 2009 Recovery Accountability and Transparency Board and Independent Advisory Panel75
ARRA, sometimes referred to as the Recovery Act, provides for a number of oversight mechanisms. One is the Recovery Accountability and Transparency Board, established "to coordinate and conduct oversight of covered funds to prevent fraud, waste, and abuse." The board's membership includes ten specified inspectors general and "any other Inspector General designated by the President from any agency that expends or obligates covered funds." The President is authorized to appoint the chairperson of the board, to be selected from among three officers. The three are: the OMB deputy director for management, a federal officer who already has been confirmed by the Senate for another position, or an individual who would have to be confirmed by the Senate. (The President selected the sitting Inspector General of the Department of the Interior (a confirmed position) to be chairperson.)
The Board may request information and assistance from any agency or entity of the federal government. Supplementing this, the Board is given the same powers of inspectors general operating under the IG Act of 1978, as amended. It may conduct public hearings; enter into contracts as may be necessary to carry out its duties; and transfer funds for expenses to support administrative services and audits, reviews, or other oversight activities.
The Board is further empowered to "conduct its own independent audits and reviews relating to covered funds and collaborate on audits and reviews" with any inspector general. In addition, the board's detailed audit and review functions pertain to the following: covered funds for relevant purposes; the reporting and competition requirements associated with contracts and grants to determine whether these meet applicable standards; the adequacy and training of personnel to oversee the covered funds; and the extent of appropriate mechanisms for interagency collaboration, including coordinating and collaborating with the Council of the Inspectors General on Integrity and Efficiency. ARRA also directs the board to make recommendations to agencies with regard to preventing waste, fraud, and abuse of the covered funds. An agency must respond within 30 days of the receipt of the recommendations, stating whether it agrees or disagrees and whether any actions will be taken to implement them.
The Board is also authorized to issue a variety of reports to the President and Congress. These include "flash reports" on potential management and funding problems that require immediate attention; quarterly reports summarizing the findings of the Board and inspectors general of agencies; annual reports consolidating the quarterly reports; and such other reports that the Board deems appropriate. In general, all of these reports are to be made available to the public on the Board's website. The website, which is to be "user-friendly" and "public-facing," is to provide links to related websites as well as materials and information regarding the act itself; accountability matters, including findings of the Board's and IGs' audits; data on contracts and grants relating to covered funds; the use and allocation of covered funds by each federal agency; data on relevant economic, financial, grant, and contract information to enhance public awareness of the use of the covered funds; and to the extent practical, job opportunities afforded by the program and how to access these on appropriate federal, state, and local websites and locations.
Supplementing the Board is a Recovery Independent Advisory Panel. Also established by ARRA, it is composed of five members appointed by the President, based on relevant expertise. The panel is to "make recommendations to the Board on actions the Board could take to prevent waste, fraud, and abuse relating to covered funds." The panel is authorized to hold hearings and to "secure directly from any agency such information as the Panel considers necessary to carry out" its duties.
Federal Information Security Management Act (FISMA) Independent Evaluations76
FISMA directs that "each year, each agency shall have performed an independent evaluation of the information security program and practices of that agency to determine the effectiveness of such program and practices."77 The Director of OMB—except for "national security systems"—is to oversee the development of the relevant information security policies, principles, standards, and guidelines; review and (dis)approve agency security programs; and take certain actions to ensure compliance with relevant policies, standards, and principles.78 Under the statute, the Director—based on guidelines and standards developed by the National Institute of Standards and Technology (NIST)—is specifically authorized to issue information security guidelines and standards, including minimum standards, which "shall be compulsory and binding."79
The statute further stipulates that the "annual evaluation required by this section shall be performed by the Inspector General or an external independent auditor," as determined by the Inspector General of the agency" or by the head of the agency if an IG has not been established by law. 80 Such an independent evaluation is to include (1) a testing of the effectiveness of information security control techniques and (2) an assessment (made on the basis of the results of the testing) of compliance with the requirements. 81 These annual reviews are to include the evaluator's findings and recommendations, which are communicated to the agency head; he or she then submits the results of the reviews to the Director of OMB, who, in turn, "shall summarize the results of the evaluations" and report these to Congress.82
Department of Defense Independent Study on Post-Traumatic Stress Disorder Efforts83
Under the National Defense Authorization Act for FY2010, the Secretary of Defense, in consultation with the Secretary of Veterans Affairs, is to provide for an independent study on the treatment of post-traumatic stress disorder. The study is to be "conducted by the Institute of Medicine of the National Academy of Sciences or such other independent entity as the Secretary shall select for purposes of the study."84
The legislation also spells out certain requirements of the study. These include a listing of each operative program and method available for the prevention, screening, diagnosis, treatment, or rehabilitation of the disorder; the status of studies and clinical trials involving innovative treatments; a description of each treatment program and a comparison of methods among them, at specified locations; current and projected future annual expenditures by DOD and VA in this matter; and a description of gender-specific and racial and ethic group-specific mental health treatment and services available for members of the Armed Forces. The entity conducting the study is to submit an initial report, due on July 1, 2012, to the Secretaries of Defense and of Veterans Affairs and appropriate congressional committees (specifically, the House and Senate Committees on Armed Services, Appropriations, and Veterans' Affairs; the House Committee on Energy and Commerce, and the Senate Committee on Health, Education, and Welfare). Responses by the Secretaries to the report—including any recommendations for on the treatment of the disorder based on the report—are required six months later. An updated report and responses are to be submitted by July 1, 2014, and January 1, 2015, respectively.
Defense Science Board Independent Assessment of Improvements in Service Contracting85
The National Defense Authorization Act for FY2010 provided that "the Under Secretary of Defense for Acquisition, Technology, and Logistics shall direct the Defense Science Board [DSB] to conduct an independent assessment of improvements in the procurement and oversight of services by the Department of Defense."86 The assessment is to cover the quality and completeness of guidance relating to the procurement of services, the extent to which best practices are being developed for setting requirements, the contracting approaches and types used for the procurement of services, and whether effective standards to measure performance have been developed. The Under Secretary, who receives the DSB study, is "to submit to the congressional defense committees a report on the results of the assessment, including such comments and recommendations as the Under Secretary considers appropriate," by a specified date (March 10, 2010, following the law's enactment on October 28, 2009).87
National Academy of Sciences Review of National Security Laboratories88
The National Defense Authorization Act for FY2010 also directs the Secretary of Energy to "enter into an agreement with the National Academy of Sciences to conduct a study of" three specified laboratories.89 The study is to include for each laboratory an evaluation: of the quality of the scientific research and the engineering being conducted there, of the criteria used to assess the scientific research and engineering, of the relationship between the quality of the science and engineering and the contract for managing and operating the laboratory, and of the management of work conducted by the laboratory for entities other than the Department of Energy. The NAS is "to submit to the Secretary of Energy a report containing the results of the study and any recommendations resulting from the study."90 By a specified date (January 1, 2011, following the October 28, 2009, enactment), the Secretary of Energy is to submit to the appropriate committees of Congress the NAS report and "any comments or recommendations of the Secretary with respect to that report."91
Federal Transit Administration State Safety Oversight Program for Rail Transit92
The Federal Transit Administration (FTA), a part of the Department of Transportation (DOT), oversees the safety and security of rail transit agencies which operate "rail fixed guideway mass transportation systems" (e.g., metrorail and subways) and which receive federal funds.93 Part of the process includes an oversight body for each jurisdiction. The program is designed, according to a GAO summary, "as one in which FTA, other federal agencies, states, and rail transit agencies collaborate to ensure the safety and security of rail transit systems."94 In most cases, each rail transit agency exists within a state, although a few extend to multi-state and other jurisdictions in such locales as the Washington, DC metropolitan area (which includes the District of Columbia, Maryland, and Virginia). Under federal law, each state or covered area
designates a State authority as having responsibility—(A) to require, review, approve, and monitor the carrying out of each safety plan; (B) to investigate hazardous conditions and accidents on the systems; and (C) to require corrective action to correct or eliminate those conditions (49 U.S.C. 5330(c)).
As an enforcement mechanism, the DOT Secretary is authorized to withhold funds if a state does not comply with these responsibilities.
The FTA program requires each state or other appropriate jurisdictions to establish an oversight body to oversee the safety and security of its system. These oversight units are responsible for developing a program standard that transit agencies must meet and for reviewing performance of the transit agencies against that standard. The oversight agency safety and security review lays out the process and criteria to be used, at least every three years, in conducting a complete review of each affected rail transit agency's implementation of the plan.
The FTA rail safety oversight program, operating through state and local oversight entities, involves
A 2006 GAO report, however, found deficiencies in the system, including a failure to meet the three-year review schedule, to develop performance goals for the program, and to be able to track performance.95 In addition, GAO reported that expertise varied across the oversight entities, which, in some cases, suffered from inadequacies in funding and qualified staff.96
Reinforcement of some of these and other findings came about after a serious accident on the Washington Metropolitan Area Transit Authority (WMATA) Metrorail system in mid-2009, which revealed short-comings of its Tri-State Oversight Committee (TOC). The Committee, for instance, lacked its own e-mail address, mailing address, telephone number; more importantly, it had limited resources, staff skills, and expertise. Additional problems included ineffective and inadequate communications with the WMATA and the absence of a process to evaluate corrective action plans and alternatives to TOC proposals.97 These defects and deficiencies, in turn, generated plans from the administration and proposals from legislators to correct them, not only in the TOC but throughout the federal transit safety system.98
Forensic Science Study Conducted by an Independent Forensic Science Committee Established by the National Academy of Sciences99
The Science, State, Justice, Commerce, and Related Agencies Appropriations Act of 2006 authorized the "National Academy of Sciences [NAS] to conduct a study of forensic science, as described in the Senate report," accompanying the legislation.100 The Senate report, which authorized NAS to establish an "independent Forensic Science Committee," recognized that
there exists little to no analysis of the remaining needs of the [criminal justice and law enforcement] community outside of the area of DNA. Therefore .... the Committee directs the Attorney General to provide [funds] to the National Academy of Sciences to create an independent Forensic Science Committee. This Committee shall include members of the forensics community representing operational crime laboratories, medical examiners, and coroners; legal experts; and other scientists deemed appropriate.101
The Senate report then set forth a series of charges to the Committee. These included assess the present and future resource needs of the forensic science community, make recommendations for maximizing the use of forensic science techniques, identify potential scientific advances, make recommendations for programs that will increase the number of qualified forensic scientists and others related professions, disseminate best practices and guidelines concerning the collection and analysis of forensic evidence, and examine the role of forensic science in homeland security.102
The NAS follow-up study reported on the members of the Committee; participants in hearings and meetings; literature reviewed; issues covered; and findings, conclusions, and recommendations.103
Improving America's Schools Act of 1994 Independent Evaluation104
This enactment establishes or expands on a number of educational programs, some of which are required to undergo an independent evaluation. The portion of the act dealing with the "Even Start Family Literacy Programs," for instance, directs the Secretary of Education to
provide for an independent evaluation of programs assisted under this part—(1) to determine the performance and effectiveness of programs assisted under this part; and (2) identify effective Even Start programs assisted under this part that can be duplicated and used in providing technical assistance to Federal, State, and local programs (Sec. 1209).
No Child Left Behind (NCLB) Act of 2001 Independent Evaluations, Reviews, and Studies, as well as Peer Reviews105
The NCLB Act includes numerous provisions for program evaluations, often done by state agencies involved in the programs, as well as for various independent evaluations, reviews, and studies. In some of these, the Secretary of Education:
Agricultural Research, Extension, and Education Reform Act of 1998 Cooperative State Research, Education, and Extension Service Peer and Merit Reviews and Advisory Board Review106
This 1998 act establishes several review mechanisms connected with various grant programs. One calls for the Secretary of the Department of Agriculture (USDA) to "establish procedures that provide for scientific peer review of each agricultural research grant administered on a competitive basis," by the USDA Cooperative Extension Service. These procedures are to include "a review panel [which] shall verify, at least every 5 years, that each research activity of the Department and research conducted under each research program of the Department has scientific merit." The enactment directs the review panel to consider "the scientific merit and relevance of the activity or research .... and the multistate significance of the activity or research." To help ensure its expertise and independence, the panel is to be "composed of individuals with scientific expertise, a majority of whom are not employees of the agency whose research is being reviewed," and who, to the maximum extent practicable, are to be selected from colleges and universities.
As part of the consideration of the scientific merit for each research activity, the enactment sets up procedures for a "merit review of each agricultural extension or education grant administered on a competitive basis," by the Cooperative Extension Service. To be eligible for a grant, relevant institutions are to "establish a process for merit review of the activity and review the activity in accordance with the process."
The results of the panel reviews are to be submitted to an Advisory Board, which is to review annually the relevance of priorities for such programs and adequacy of funding for them. The Secretary is required to consider the results of the Advisory Board's review "when formulating each request for proposals, and evaluating proposals" involving such programs.
Fund for the Improvement of Post-Secondary Education (FIPSE) Independent Outside Evaluator107
FIPSE, operating under a project director, provides grants connected with relevant projects. An independent outside evaluator (IOE) is called for in the legislation. Hired by the project director, the IOE "must be someone who does not stand to gain personally or professionally from the project results." According to the department's guidance, the IOE is to attend the project director's meeting on evaluating a FIPSE grant and "to assist the Director in completing the initial evaluation plan/chart, due three months after the start of the grant." The IOE is also to assist the director in a number of matters, including offering advice about: which project objectives would lend themselves to measurement and evaluation, which baseline data should be collected, which measurement instruments could be used, what data might be collected from a possible comparison or control group, and how the director might disseminate evaluation results to interested parties. The independent outside evaluator is also to assist the project director in designing the evaluation instruments and to write the evaluation reports.
The Carl D. Perkins Vocational and Technical Education Assistance Act Independent Advisory Panel and Independent Evaluation108
This legislation creates an Independent Advisory Panel (IAP)—appointed by the Secretary of Education and consisting of representatives from a broad range of experts and affected parties—to advise him or her on the implementation and assessment of the programs authorized by the act. The Secretary is also authorized to collect relevant information from states and localities which can be used in an evaluation. To assist in this, the Secretary "shall provide for the conduct of an independent evaluation and assessment of vocational and technical programs under this Act through studies and analyses conducted independently through grants, contracts, and cooperative agreements that are awarded on a competitive basis." The enactment also details the contents of such an assessment. These extend to: the efforts and effects of state, local, and tribal entities on such programs; impact of federal expenditures that address program improvements in relevant educational programs; preparation and qualifications of teachers in the fields; academic and employment outcomes of the education; employer involvement and satisfaction with such educational programs; use and impact of educational technology and "distance learning" in the field; and effect of state adjusted levels of performance and state levels of performance on the delivery of relevant services.
The Secretary is also to submit to specified congressional committees an interim report on the assessment and a final report summarizing all studies and analyses related to the assessment. These reports "shall not be subject to any review outside the Department of Education" prior to their submission to Congress. But "the President, Secretary, and advisory panel .... may make such additional recommendations to Congress with respect to the assessment" as each determines appropriate.
Occupational Information To Be Collected by the Social Security Administration (SSA) Independent Evaluation109
Under this highly specialized, narrowly focused request, the SSA sought the services of an "Independent Evaluator (IE)" regarding "pre-award and post-award evaluations of occupational information and methodology employed by private sector entities." The IE was expected to help revise and update existing coverage published by the Department of Labor. Both pre-award and post-award evaluation services and requirements are detailed in the SSA request itself and in its attachments.
Acknowledgments
Additional CRS contributors to the research for this report are Kim Klarman and Pervaze Sheikh.
| 1. |
For discussion, see CRS Report RL33301, Congress and Program Evaluation: An Overview of Randomized Controlled Trials (RCTs) and Related Issues, by [author name scrubbed], [author name scrubbed], and [author name scrubbed]. |
| 2. |
The Library of Congress's catalogues list a number of titles using "independent evaluation" and "independent evaluator." But these pertain mostly to reports on and by such units in ad hoc, highly specialized efforts, undertaken by foreign governments, international organizations, and federal, state, and local governments in the United States. |
| 3. |
See discussion in Eleanor Chelimsky, "The Coming Transformations in Evaluation," in Eleanor Chelimsky and William R. Shardish, eds., Evaluation for the 21st Century: A Handbook (Thousand Oaks, CA: Sage, 1997), pp. 18-22. |
| 4. |
William R. Shadish and Jason K. Luellen, "History of Evaluation," in Sandra Mathison, ed., Encyclopedia of Evaluation (Thousand Oaks, CA: Sage, 2005), p. 183. |
| 5. |
William R. Shadish Jr., Thomas D. Cook, and Laura C. Leviton, Foundations of Program Evaluation: Theories of Practice (Newbury Park, CA: Sage, 1991), p. 22. For more on evaluation and related concepts, see CRS Report RL30240, Congressional Oversight Manual, by [author name scrubbed], et al.; CRS Report RL33301, Congress and Program Evaluation: An Overview of Randomized Controlled Trials (RCTs) and Related Issues, by [author name scrubbed], [author name scrubbed], and [author name scrubbed]; and U.S. Congress, Senate Committee on Government Operations, Legislative Oversight and Program Evaluation: A Seminar Sponsored by the Congressional Research Service, committee print, 94th Cong., 2nd sess. (Washington: GPO, 1976). A more recent manifestation of program evaluation's anticipated benefits appeared in the President's FY2011 budget: U.S. Executive Office of the President, Office of Management and Budget (hereafter OMB), Budget of the U.S. Government, Fiscal Year 2011, Analytical Perspectives (Washington, GPO, 2010), pp. 91-92.). |
| 6. |
Illustrative of this is U.S. Department of Energy (DOE), Overview of Evaluation Methods for R&D Programs: A Directory of Evaluation Methods Relevant to Technology Development Programs (Washington, DOE, 2007). |
| 7. |
Coverage of these and many other statutes is in CRS Reports CRS Report RL30795, General Management Laws: A Compendium, by [author name scrubbed], et al. (pdf), pp. 72-76. |
| 8. |
The IG Act (5 U.S.C. Appendix)—establishing offices of inspectors general initially in 1978—is among other earlier enactments which allow for some type of program evaluation but which may not have been pursued extensively until the 1990s or later. Until then, few of the offices of inspector general (OIGs) conducted inspections/evaluations (I&E), despite implied authority to do so. A 1990 study, for instance, found that only eight of the more than 60 OIGs at the time had an inspection activity similar to program evaluation, although its performance was viewed as a "a definite and increasing trend." Michael Hendricks, Michael F. Mangano, and William C. Moran, eds., Inspectors General: A New Force in Evaluation (San Francisco, Jossey-Bass Inc., 1990), pp. 1 and 77. By 2007, the number had increased substantially, according to a survey of the 57 offices directly under the IG Act at the time (36 of which responded). A majority of the offices—31—reported that they conducted I&E work. Among these, 25 had staff dedicated to I&E, either in stand-alone units or within other units (e.g., audit); and two OIGs had set up multiple I&E units. PCIE/ECIE, 2007 Survey of Inspection and Evaluation Units in the Federal Inspector General Community, pp. i and 1, at http://www.ignet.gov. Other comparisons further indicate the expansion of inspection and evaluation within the IG community over the recent past. From 1999 to 2007, the number of separate I&E units grew from 15 to 25; the average annual budget for them rose from $2.42 million to $3.95 million; and the number of full-time equivalent employees increased from 415 to 648. Ibid., p. i. The belated adoption of inspection and evaluation in the IG community is reflected in other developments. It was not until 1993, importantly, that the coordinative councils of the inspectors general issued the first set of quality standards for inspections. PCIE/ECIE, Quality Standards for Inspections (revised), January 2005, cover letter, at http://www.ignet.gov. At about the same time, the IGs approved a vision statement that reflected a broad change along the lines recommended by President William Clinton's National Performance Review (NPR), suggesting an expanded role for program evaluation. A Government Accountability Office (then General Accounting Office; hereafter GAO) study summarized the NPR recommendation for the IGs to change "the focus of IGs from compliance auditing to evaluating management control systems and recasting their method of operations to be more collaborative and less adversarial. In January 1994, the IGs adopted an 'Inspectors General Vision Statement' that states: "We are agents of positive change striving for continuous improvement in our agencies' management and program operations and in our own offices." GAO, Federal Inspectors General: An Historical Perspective, GAO/T-AIMD-98-146, April 21, 1998, p. 4. The original IG Act of 1978, however, directed the statutory IGs to appoint assistant IGs only for audit and for investigation and not for inspection or program evaluation. A provision for all statutory IGs to do so in their offices has not been added by subsequent amendments. Nonetheless, the 2008 amendments to the IG act call on a new Council of the Inspectors General for Integrity and Efficiency (CIGIE), composed of the statutory IGs and other relevant offices, to "develop plans for coordinated, Governmentwide activities that ... promote economy and efficiency in Federal programs and operations, including interagency and interentity audit, investigation, inspection, and evaluation programs and projects ... that exceed the capability or jurisdiction of an individual agency or entity " (Inspector General Act Reform Act of 2008, P.L. 110-409, 122 Stat. 4308). The same enactment increases the prominence of program evaluations by requiring that the IG semiannual reports include "inspection reports, and evaluation reports" (122 Stat. 4315). |
| 9. |
GPRA or the Results Act provides that each agency develop a mission statement, on which is based a five-year strategic plan setting long-range goals, which can be adjusted to meet new demands and changing conditions; an annual plan for putting the goals into effect, and a yearly follow-up evaluation. The Results Act is designed to recognize the importance of outcomes (that is, the measured impact and effectiveness of a program) as opposed to outputs (that is, for example, the amount of money funding a project, the issuance of new regulations, or the number of grants). A bill introduced in 2010—the Government Efficiency, Effectiveness, and Performance Act of 2010 (H.R. 2142, 1111h Cong., 2nd sess.)—would build on GPRA in several ways. One provision, for instance, would require quarterly performance assessments of government programs in order to evaluate their performance and improvement. See introductory remarks by Representatives Chaffetz. Cuellar, Platts, and Watson, in Congressional Record, June 16, 2010, vol. 156, pp. H4553-H4560; and CRS congressional distribution memorandum, Analysis of Subcommittee-Reported H.R. 2142 (111th Congress) and Related Issues, by [author name scrubbed] (available from author). |
| 10. |
Importantly under the Inspector General Act of 1978, OIGs are intended to be independent units, even though each is located within a parent "establishment" or "designated federal entity." Supporting their independence, inspectors general operate only under the "general supervision" of the agency head; and except in a few cases and under specified circumstances, the head cannot prevent or halt an IG investigation or audit and must transmit IG reports, including recommendations for corrective action, to Congress unaltered. In addition, IGs are prohibited from undertaking program operating responsibilities. Other supports include notification to Congress of their initial budget requests, along with any changes made by agency administration or OMB, and an IG statement if he or she considers any reductions to be harmful to carrying out the OIG responsibilities. Finally, IGs in establishments are appointed by the President with Senate confirmation and in DFEs by the agency head. In both types of offices, Congress must be notified in writing of the reasons 30 days in advance of any IG's removal by the President or agency head, respectively. See also footnote 19 below. |
| 11. |
The independence of GAO, a congressional support agency, from the executive agencies it reviews, audits, investigates, or inspects has been an integral part of the Office since its establishing authority in 1921. Then known as the General Accounting Office, it "shall be independent of the executive departments and under the control and direction of the Comptroller General of the United States" (P.L. 67-13, codified at 31 U.S.C. 41). See also footnote 18 below. |
| 12. |
This absence is in contrast, for instance, to statutory IGs under the IG Act. Among other things, the act spells out key commonalities, including their appointment, removal, supervision, broad responsibilities and duties, investigative and audit powers, reporting requirements, staffing and funding, and jurisdiction. Differences among IGs under the IG Act, which are relatively few, are specified in the law for an individual office or are confined to a specific policy matter or project. |
| 13. |
GAO, Program Evaluation: OMB's PART Reviews Increased Agencies' Attention to Improving Evidence of Program Results, GAO-06-76, October 2005, p. 25. Further discussion of OMB and PART appears below. |
| 14. |
Eleanor Chelimsky, "The Coming Transformations in Evaluation," in Eleanor Chelimsky and William R. Shardish, eds., Evaluation for the 21st Century: A Handbook (Thousand Oaks, CA: Sage, 1997), pp. 18-22. |
| 15. |
AEA provides guidance for scholars and practitioners in conducting evaluations, including their potential contributions to policymaking and program implementation. AEA, Evaluation, Evaluators, and the American Evaluation Association, at http://www.eval.org/aea09.eptf.policy.handouts.pdf; and American Evaluation Association Guiding Principles for Evaluators (2004), at http://www.eval.org/GPTraining/GP%20Training%20Final/gp.principles.pdf. |
| 16. |
For example, see Gail V. Barrington, "Independent Evaluation," in Encyclopedia of Evaluation (Thousand Oaks, CA: Sage, 2005), p. 199. |
| 17. |
Joint Committee on Standards for Educational Evaluation, The Program Evaluation Standards: How to Assess Evaluations in Educational Programs (Thousand Oaks, CA: Sage, 1994). Established in 1975, the Joint Committee is comprised of 16 professional educational associations, ranging from the American Association of School Administrators, AEA, and the American Psychological Association to the National Council on Measurement in Education and the National Education Association. The Joint Committee's experience in the field of program evaluation, including its recommended standards, has focused on education programs; but these are seen as also having general applicability to other public policy areas. See Daniel L. Stufflebeam, "Joint Standards for Education Evaluation," in Encyclopedia of Evaluation, pp. 213-214. |
| 18. |
Among the many GAO products that demonstrate its long-standing interest in program evaluation are: Performance Measurement and Evaluation: Definitions and Relationships, GAO-05-739SP, May 2005; Program Evaluation: OMB's PART Reviews Increased Agencies' Attention to Improving Evidence of Program Results, GAO-06-67, October 2005; Program Evaluation: An Evaluation Culture and Collaborative Partnerships Help Build Agency Capacity, GAO-03-454, May 2003; Designing Evaluations, GAO/PEMD-10.1.4, March 1991; Case Study Evaluations, GAO-PEMD-91-10.1.9, November 1990; Prospective Evaluation Methods: The Prospective Evaluation Synthesis, PEMD-10.1.10, November 1, 1990; and The Evaluation Synthesis, PEMD-10.1.2, March 1, 1990. Bolstering GAO's involvement in this field is the 1982 codification of title 31 of the U.S. Code, Money and Finance, which provides for evaluating programs and activities of the United States Government (P.L. 97-258, 96 Stat. 893-894). Under it, the Comptroller General is to "develop and recommend to Congress ways to evaluate a program or activity" as well as to assist a requesting congressional committee to "develop a statement of legislative goals and ways to assess and report program performance related to the goals" and to "assess program evaluations prepared by and for an agency." Ibid. |
| 19. |
As noted above, IGs, operating directly under the Inspector General Act of 1978, have been given substantial independence and powers to combat waste, fraud, and abuse, including implied authority to conduct inspections, which includes program evaluation. IG inspection standards are spelled out in PCIE/ECIE, Quality Standards for Inspections, at http://www.ignet.gov. For coverage of relevant units within IG offices, see PCIE/ECIE, 2007 Survey. The IGs now also belong to a single collective council established by law. The Inspector General Reform Act of 2008 (P.L. 110-409; 122 Stat. 4302) created a Council of the Inspectors General on Integrity and Efficiency (CIGIE), which merged, along with other units, two previous IG councils: the President's Council on Integrity and Efficiency (PCIE) and the Executive Council on Integrity (ECIE). The new Council has been charged with using inspections and program evaluations, along with audits and investigations, to handle problems which exceed the capacity or jurisdiction of an individual agency (122 Stat. 4307). Reinforcing its audit and inspection/evaluation activities, CIGIE has adopted, as one of its three major goals, contributing to "government-wide improvements in program integrity, efficiency, and cost-effectiveness by providing cross-agency analysis of OIG findings and recommendations in areas of vulnerability confronting multiple government programs." Phyllis K. Fong, Inspector General, U.S. Department of Agriculture, and Chair, CIGIE, "The IG Reform Act and the New IG Council: Dawn of a New Era," Journal of Public Inquiry, Fall/Winter, 2008-2009, p. 5. CIGIE also includes a standing Committee on Inspections and Evaluation, as had its predecessors (see note 8, above). The Committee's website, incidentally, provides "related links" to AEA and GAO, at http://www.ignet.gov/pande/iande1.html. |
| 20. |
OMB, Guide to the Program Assessment Rating Tool (PART), Guidance No. 2008-01, January 2008, pp. 1, 30-36, and 59-60. For a GAO study of this part of PART, introduced in the George W. Bush Administration, see GAO, OMB's PART Reviews. In 2009, the Obama Administration developed its own approach: OMB, Increased Emphasis on Program Evaluation, Memorandum for the Heads of Executive Departments and Agencies, M-10-01, October 7, 2009. OMB expanded upon this the next year in two memoranda: OMB, Evaluating Programs for Efficacy and Cost-Efficiency, Memorandum for the Heads of Executive Departments and Agencies, M-10-32, July 29, 2010; and Performance Improvement Guidance: Management Responsibilities and Government Performance and Results Act Documents, Memorandum from Shelley Metzenbaum, OMB Associate Director for Performance and Personnel Management, June 25, 2010. |
| 21. |
AEA, Evaluation, p. 1. |
| 22. |
Ibid. |
| 23. |
Ibid., p. 4. |
| 24. |
Ibid. |
| 25. |
AEA, Guiding Principles. |
| 26. |
Ibid., pp. 2-6 (Bold in original). |
| 27. |
Barrington, "Independent Evaluation." |
| 28. |
See Stufflebeam, "Joint Committee for Education Evaluation." |
| 29. |
Joint Committee, Program Evaluation Standards, pp. 223-224. |
| 30. |
Ibid. |
| 31. |
GAO, Performance Measurement and Evaluation, p. 2. The four types are process (or implementation) evaluation, outcome evaluation, impact evaluation, and cost-benefit and cost-effectiveness analyses. Ibid., p. 4. See also footnote 18 above. |
| 32. |
P.L. 110-409, 122 Stat. 4307. |
| 33. |
PCIE/ECIE, Quality Standards for Inspections, p. i. See footnotes 8, 10, and 19 above for further discussion of the IGs' roles in evaluation and their independence. |
| 34. |
Ibid. |
| 35. |
Ibid, pp. 1-28. |
| 36. |
Ibid., cover letter. |
| 37. |
Ibid. |
| 38. |
PCIE/ECIE, 2007 Survey, cover letter and pp. i and 1. |
| 39. |
Ibid. |
| 40. |
OMB, Guide to PART, pp.30-36 and 59-60. See also, GAO. |
| 41. |
OMB, Guide to PART, pp. 30 and 60. |
| 42. |
Ibid. |
| 43. |
GAO, Program Evaluation: OMB's PART Reviews Increased Agencies' Attention to Improving Evidence of Program Results, GAO-06-76, October 2005. |
| 44. |
GAO, OMB's PART Reviews, p. 25. |
| 45. |
OMB, Guide to PART, p. 36. |
| 46. |
OMB, Increased Emphasis on Program Evaluations (2009), p. 1; and Evaluating Programs for Efficacy and Cost-Efficiency (2010), p. 1, whose single-word difference is bracketed. |
| 47. |
OMB, Increased Emphasis on Program Evaluations, Memorandum, M-10-01, October 7, 2009. |
| 48. |
OMB, Evaluating Programs for Efficacy and Cost-Efficiency (2010), p. 1. |
| 49. |
Ibid., p. 4. |
| 50. |
Ibid. |
| 51. |
An impact evaluation seeks to quantitatively estimate the extent to which a program affects an outcome of interest (e.g., the unemployment rate) compared to what would have happened without the program. Ibid. The subsequent OMB memorandum from the Director also focused on "impact evaluations, or evaluations aimed at determining the causal effects of programs or particular strategies, interventions, and within programs." OMB, Evaluating Programs for Efficacy and Cost-Efficiency (2010), p. 2. |
| 52. |
OMB, Increased Emphasis on Program Evaluations (2009) and Evaluating Programs for Efficacy and Cost-Efficiency (2010), p. 1. The goal is "that each non-security agency submit a budget request 5 percent below the agency's FY 2012 discretionary total in the FY 2011 Budget." OMB, Evaluating Programs for Efficacy and Cost-Efficiency, p. 1, which details the process for addressing the problems and strengthening program evaluation. |
| 53. |
OMB, Performance Improvement Guidance, pp. 1-2. |
| 54. |
GAO, Streamlining Government: Opportunities Exist to Strengthen OMB's Approach to Improving Efficiency, Report GAO-10-394. The major portion of this GAO performance audit was conducted from September 2008 to May 2010. Ibid., p. 5. |
| 55. |
Ibid., Summary. |
| 56. |
Ibid. |
| 57. |
Ibid., Summary and pp. 2-4. |
| 58. |
OMB, Performance Improvement Guidance, pp. 1-2. |
| 59. |
Several national academies—National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council—have been called upon to provide independent evaluations and assessments directly or to advise agency officials on these. For a listing of these charges from the 106th to the 111th Congress, see U.S. National Academies, Short Summaries of National Academies' Studies Within Public Laws by Congressional Session, available at http://www7.nationalacademies.org/ocga/Other/Short_Summaries. |
| 60. |
An illustration of this involves the National Institute of Science and Technology (NIST) and its review of the Department of Agriculture's (USDA) National Organic Program (NOP). According to USDA, the NOP has applied to NIST "for recognition of its accreditation function.... Applying for and receiving recognition will support NOP's credentials as an accrediting body and satisfy regulatory requirements to obtain peer review.... Third-party recognition is important for many of USDA's audit-based programs." Kathleen A. Merrigan, Deputy Secretary, USDA, letter, June 4, 2009, to Steve Etka, Executive Director, National Organic Coalition, Washington, DC. |
| 61. |
President Barack Obama, Executive Order 13508, Chesapeake Bay Protection and Restoration, May 12, 2009, 74 FR 23099-23104, May 15, 2009. |
| 62. |
EO 13508, Sec. 201. The federal departments and agencies represented are: the Environmental Protection Agency (EPA), whose Administrator chairs the Committee; the Departments of Agriculture, Commerce, Defense, Homeland Security, Interior, and Transportation; and such other agencies as determined by the Committee. This new development ties into long-term efforts of the Chesapeake Bay Program, a regional partnership that has existed since 1983. The Program consists of EPA, which represents the Federal government and manages the Program Office; six states (Delaware, Maryland, New York, Pennsylvania, Virginia, and West Virginia); the District of Columbia; the Chesapeake Bay Commission, a tri-state legislative body; and participating citizen advisory groups. Descriptions of actions taken by EPA, include EPA, Guidance for Federal Land Management in the Chesapeake Bay, May 12, 2010, and The Chesapeake Bay Program, available http://www/epa.gov/region03/chesapeake/indx.htm. |
| 63. |
E.O. 13508, Sec 201. |
| 64. |
Ibid., Sec. 203. As described by EPA, "biennial monitoring, tracking and reporting will be conducted to assess the effectiveness of implementation actions identified in the two-year milestones [goals and commitments by states to reduce pollution in the Bay]. These data will also be used to drive adaptive decision-making and redirect management actions." EPA, Chesapeake Bay Program, "Frequently Asked Questions," p. 7. |
| 65. |
Ibid., Sec. 206. Earlier in the executive order, the District of Columbia is identified separately from Chesapeake Bay area States. Later, the Committee is directed "to consult extensively with the States of Virginia, Maryland, Pennsylvania, West Virginia, and Delaware, and the District of Columbia." Ibid. Sec. 204. Because the District of Columbia is not a state, it does not technically fall under the rubric in Section 206, which calls for "collaboration with State agencies." By comparison, H.R. 1053 Sec. 4(3), explicitly includes the District of Columbia under its definition of "States." |
| 66. |
S. 1816, 111th Cong., 1st sess., introduced by Senator Benjamin Cardin for himself and others. The bill and supportive statements are at Congressional Record, vol. 155, October 20, 2009, pp, S10573-S10581. For additional comments, see "Cardin, O'Malley, Cummings, Van Hollen and regional partners detail new legislation to restore the Chesapeake Watershed," press release, October 20, 2009; and letter of support to Honorable Benjamin L. Cardin from Maryland Governor Martin O'Malley, Virginia Governor Timothy M. Kaine, and District of Columbia Mayor Adrian M. Fenty, October 20, 2009, both available at http://www.cardin.senate.gov/news/indx/cm. |
| 67. |
H.R. 1053, 111th Cong., 1st sess., an Amendment in the Nature of a Substitute to H.R. 1053, offered by Representative Wittman, as agreed to by the House Committee on Natural Resources, July 29, 2009. For a press account, see "CQ Committee Coverage," Congressional Quarterly, July 29, 2009, p. 2, at http://www.cq.com/find.do?dataSource=committeemarkups&queryFragment. |
| 68. |
One CRS study describes a crosscut budget as follows: "At its most basic level, a crosscut budget is often used to present budget information from two or more agencies whose activities are targeted at a common policy goal or related policy goals .... In the context of ecosystem restoration, a crosscut budget is typically a document that organizes and reports the activities and funding of several entities within the same broad initiative in a way that 'cuts across' organizational boundaries." CRS Report RL34329, Crosscut Budgets in Ecosystem Restoration Initiatives: Examples and Issues for Congress, by [author name scrubbed] and [author name scrubbed]. Analyzing crosscut budgeting in a broad context, including its possible policy-making and power-broker effects, Allen Schick emphasized its role as an action-forcing mechanism, designed to compel agencies to cooperate in furtherance of a shared policy responsibility. Allen Schick, "The Coordination Option," in Peter Szanton, ed., Federal Reorganization: What Have We Learned? (Chatham House Publishers, Inc., Chatham, New Jersey, 1981), pp. 99-101. |
| 69. |
P.L. 106-541; 114 Stat. 2595-2596 and 2691. |
| 70. |
114 Stat. 2691. |
| 71. |
114 Stat. 2595-2596. |
| 72. |
33 U.S.C. 2731. |
| 73. |
33 U.S.C. § 2761. |
| 74. |
33 U.S.C. § 2343. |
| 75. |
P.L. 111-5, 123 Stat. 289-296. For an overview, see CRS Report R40572, General Oversight Provisions in the American Recovery and Reinvestment Act of 2009 (ARRA): Requirements and Related Issues, by [author name scrubbed]. |
| 76. |
Codified at 44 U.S.C. Chapter 35. FISMA replaced the National Defense Authorization Act for FY2001 (P.L. 106-398, Title X, Subtitle G, sec. 1061; 114 Stat. 1654A-271-1654A-272), which had already contained requirements for annual independent evaluations along the same lines. |
| 77. |
44 U.S.C. 3545(a)(1). |
| 78. |
44 U.S.C. 3543(a) and 3544(b). Certain national security systems, however, fall under the jurisdiction and authority of the Director of National Intelligence, Director of the Central Intelligence Agency, and Secretary of Defense. 44 U.S.C. 3543(b)-(c) and 3545(g)(3) and 40 U.S.C. 11331(b)(2). Another provision adds that "standards and guidelines for national security systems .... shall be developed, promulgated, enforced, and overseen as otherwise authorized by law and as directed by the President." 40 U.S.C. 11331(b)(1)(C). An earlier version, incidentally, had specified that the "President may disapprove or modify the standards and guidelines." 40 U.S.C. 11331, note on "Amendments." For additional information, see U.S. Department of Commerce, NIST, Information Security: Recommended Security Controls for Federal Information Systems and Organizations, NIST Special Publication 800-53, Revision 3 (August 2009). The NIST report recognizes that such "standards and guidelines shall not apply to national security systems without the express approval of appropriate federal officials exercising policy authority over such systems." Ibid., p. iii. |
| 79. |
44 U.S.C. 3543 (a)(3) and 40 U.S.C. 11331(b) and (d); and NIST, Information Security, with the exception of national security information systems, as previously noted. |
| 80. |
44 U.S.C. 3545(b). |
| 81. |
44 U.S.C. 3545(a)(2)(A)-(B). |
| 82. |
44 U.S.C. 3545(g) and 3543(a)(8). |
| 83. |
P.L. 111-84, sec. 726 (123 Stat. 2395-2397). |
| 84. |
Ibid. |
| 85. |
P.L. 111-84, sec. 802 (123 Stat. 2400-2401). The Defense Science Board (DSB), considered a federal advisory commission, was established by the Secretary of Defense in 1956, in response to a recommendation of the Hoover Commission. The Board, which has 32 members plus seven ex officio members, operates through various task forces consisting of Board members and other experts/consultants, to conduct studies at the direction of various DoD officials. Under its charter, the DSB is to provide "independent advice and recommendations on scientific, technical, manufacturing, acquisition process, and other matters of special interest to the Department of Defense." DSB provides such advice to the Secretary of Defense and other Department officials by way of formal briefings and written reports. DOD, Defense Science Board, About the DSB, History, and Charter: Defense Science Board, available at http://www.acq.osd.mil/dsb/htm. |
| 86. |
P.L. 111-84, sec. 802(b) (123 Stat. 2400-2401). |
| 87. |
Ibid. Sec. 803(c) (123 Stat. 2401). |
| 88. |
P.L. 111-84, Sec 3131 (123 Stat. 2711-2712). Under Sec. 3131(c) (123 Stat. 2711), "the Secretary of Energy shall, in consultation with the Secretary of Defense and the Director of National Intelligence, ensure that the National Academy of Sciences receives full and timely cooperation from the Department Energy, Department of Defense, and the intelligence community ... in conducting the study." |
| 89. |
Ibid. |
| 90. |
Ibid. |
| 91. |
Ibid. |
| 92. |
49 U.S.C. 5330 and 49 C.F.R. 659. |
| 93. |
49 U.S.C. 5330(a)-(b). |
| 94. |
GAO, Rail Transit: Additional Federal Leadership Would Enhance FTA's State Safety Oversight Program, GAO-06-821, July 2006, p. 3,. For further coverage, see CRS Report R40688, The Federal Role in Rail Transit Safety, by [author name scrubbed] and [author name scrubbed]. The authors recognize that "rail transit operations are an inherently local activity, and the federal government has limited responsibility for the safety of rail transit operations." Ibid., Summary. |
| 95. |
GAO, Rail Transit, Highlights and p. 1. |
| 96. |
Ibid. Additional analysis of the FTA safety program is in CRS Report R40688, The Federal Role in Rail Transit Safety. |
| 97. |
The FTA findings and recommendations along with the TOC response appear at DOT, FTA, Tri-State Oversight Committee and the Washington Metropolitan Area Transit Authority: Final Audit Report, March 4, 2010 (Washington, DOT/FTA, 2010); and TOC, TOC Response to FTA Final Audit Report, May 4, 2010 (Washington, WMATA, 2010). For additional coverage, see U.S. Congress, House Committee on Oversight and Government Reform, The Washington Metro System: Safety, Service, and Stability, hearing, 111th Cong., 2nd sess., April 21, 2010, at http://oversight.house.gov/indx.php?option=com_content&task=view&id=4892&Itemid=2. |
| 98. |
For proposals from legislators and the administration, which, in part, are designed to improve oversight, see National Metro Safety Act, H.R. 3338, and S. 1506, 111th Congress; and U.S. Congress, House Committee on Transportation and Infrastructure, Subcommittee on Highways and Transit, Public Transit Safety: Examining the Federal Role, hearing, 111th Cong., 1st sess., December 8, 2009, at http://transportation.house.gov/hearings/hearingDetail.aspx?NewsID=1060. |
| 99. |
P.L. 109-108, 119 Stat. 2290. |
| 100. |
Conference Committee report, House Rept. 109-272, 109th Cong., 1st sess., p. 121. The NAS study—Strengthening Forensic Science in the United States: A Path Forward—is available at http://books.nap.edu/catagol/12598.html. |
| 101. |
Senate rept. 109-88, 109th Cong., 1st sess., p. 46. |
| 102. |
Ibid. |
| 103. |
NAS, Strengthening Forensic Science. pp. 1-7. |
| 104. |
P.L. 103-382, Sections 1202 and 1209; 108 Stat. 3578 and 3584. |
| 105. |
P.L. 107-110; 115 Stat. 1555, 1566, 1596-1597, 1890, and 1931. |
| 106. |
P.L. 105-185; 112 Stat. 527-528. |
| 107. |
Department of Education, Guidance on the Role of the Independent Outside Evaluator, at http://www.ed.gov/print/about/offices/list/ope/fipse/outsideevaluator.html, July 23, 2009. |
| 108. |
P.L. 105-332, 112 Stat. 3090-3093. |
| 109. |
SSA, Request for Information, Solicitation Number: Reference-Number-IE-01, June 23, 2004. |