<font size="+1">List of Tables</font>
In September 2003, the Office of Management and Budget (OMB) published a proposed bulletin on "Peer Review and Information Quality" in the Federal Register that sought to establish a process by which all "significant regulatory information" would be peer reviewed. The scope of the proposed bulletin was very broad, covering virtually all agencies and defining regulatory information as "any scientific or technical study that ... might be used by local, state, regional, federal and/or international regulatory bodies." Such information would be subject to peer review if the agency could determine that it could have a "clear and substantial impact on important public policies or important private sector decisions" when disseminated. The proposed bulletin placed additional peer review requirements on "especially significant regulatory information," and said agencies were required to notify OMB in advance of any studies that might require peer review and how any such reviews would be conducted.
The proposed bulletin aroused controversy, with some observers expressing concern that it could create a centralized peer review system within OMB that would be vulnerable to political manipulation or control by regulated entities. OMB received nearly 200 comments on the proposal, and published a "substantially revised" peer review bulletin in April 2004 that was broader in scope than the proposed bulletin in that it applied to "influential scientific information" (which includes, but is not limited to, regulatory information) and "highly influential scientific assessments." However, agencies were given substantial discretion to decide whether information is "influential" and therefore requires a peer review. The revised bulletin also allowed agencies to use the National Academy of Sciences for peer reviews or to use other procedures that had been approved by OMB. It also provided exemptions for certain classes of information, such as information related to national security, products by government-funded scientists that are not represented as views of a federal agency, and routine statistical information. However, OMB retained significant authority to decide when information is "highly influential" (and, therefore, requires more specific peer review procedures) and to approve alternative peer review procedures.
OMB received more than 50 comments on the revised peer review bulletin, many of which were supportive of the changes made to the proposal. However, some commenters believed the changes did not go far enough, while others believed that OMB had significantly weakened the bulletin. In January 2005, OMB published a final version of the bulletin with what it described as "minor revisions" to the version published in April 2004 (e.g., requiring agencies to disclose the identities of peer reviewers and to prepare an annual report on their peer review activities). A number of issues regarding the implementation of the bulletin remain unclear (e.g., how much discretion agencies will be given to decide when and what kind of peer review is required). This report will be updated when any further revisions to the bulletin are published or other significant events occur.
On September 15, 2003, the Office of Management and Budget (OMB) in coordination with the Office of Science and Technology Policy (OSTP), published a proposed bulletin in the Federal Register on "Peer Review and Information Quality." The bulletin, if made final, would have established a process by which all "significant regulatory information" and "especially significant regulatory information" would be peer reviewed.(1) OMB described the term "peer review" in this context as "a scientifically rigorous review and critique of a study's methods, results, and findings by others in the field with requisite training and expertise." The proposed bulletin placed additional peer review requirements on "especially significant regulatory information," and said agencies were required to notify OMB in advance of any studies that might require peer review. The scope of the proposed bulletin was very broad, covering virtually all agencies and defining "regulatory information" as "any scientific or technical study that ... might be used by local, state, regional, federal and/or international regulatory bodies." OMB indicated it was issuing the bulletin because agencies' peer review practices were inconsistent, and government-wide standards for peer review would make regulatory science more competent and credible.
The proposed bulletin aroused substantial controversy, with some observers expressing concern that it could create a centralized peer review system within OMB that would be vulnerable to political manipulation or control by regulated entities. OMB received nearly 200 comments on the proposal, including comments from Members of Congress, trade associations, public interest groups, and recognized experts in the field of peer review and scientific research. As a result of those comments, OMB (again in consultation with OSTP) published what it described as a "substantially revised" peer review bulletin in the Federal Register on April 28, 2004.(2) In some ways, the revised bulletin was broader than its predecessor. For example, instead of focusing on "significant" and "especially significant regulatory information," the revised bulletin centered on "influential scientific information" (which includes, but is not limited to, regulatory information) and "highly influential scientific assessments." In other ways, though, the revised bulletin was less inclusive and directive. For example, it gave agencies more discretion to determine when information required a peer review, and when the more detailed review requirements for "highly influential" information were applicable. Also, unlike the proposed bulletin, the revised bulletin did not exclude individuals from being peer reviewers if they had received research grants from the agency disseminating the information being peer reviewed. OMB again requested comments from the public on the bulletin.
On December 15, 2004, OMB published a final version of the peer review bulletin on its website.(3) The final bulletin was published in the Federal Register on January 14, 2005.(4) OMB said this version reflects "minor revisions" made in response to more than 50 comments from the public on the revised bulletin. For example, the final bulletin requires agencies to disclose the names of peer reviewers to the public and adds an annual reporting requirement to allow OMB to track how agencies are using the bulletin. However, agencies are still afforded substantial discretion to determine when and what type of peer review is required. OMB also retains substantial discretion in certain areas.
This report briefly describes OMB's September 2003 peer review proposal, the concerns that were raised about that proposal in the public comments and elsewhere, and OMB's April 2004 revised bulletin. It then describes comments on the revision and OMB's final bulletin. First, however, the report provides some background on peer review, particularly in the context of regulatory policy and recent efforts to improve the quality of regulatory information.
There is no official, government-wide definition of "peer review," even in the OMB bulletins.(5) Agency-specific definitions and descriptions of the term differ, but all of them contain the basic concept of a review of scientific or technical merit by individuals with both sufficient technical competence and no unresolved conflicts of interest. In its final bulletin, OMB states that peer review "typically evaluates the clarity of hypotheses, the validity of research design, the quality of the data collection procedures, the robustness of the methods employed, the appropriateness of the methods for the hypotheses being tested, the extent to which the conclusions follow from the analysis, and the strengths and limitations of the overall product."(6)
Some type of peer review has been used for centuries within the scientific community to judge the quality of science. Peer review can take many different forms, and is used for a variety of purposes. For example, it is used commonly by federal agencies to evaluate research proposals, and plays a major role in funding decisions. In these cases it is often part of a broader category of evaluation known as merit review. Peer review is also the usual method by which the editors of scientific publications evaluate proposed research reports. It even plays an important role in many research institutions, including many government agencies, in decisions about retention and promotion for individual scientists and in reviewing research programs. A February 1999 report from the Committee on Science, Engineering, and Public Policy concluded that expert review (the most common form of which is peer review) is the most effective means of evaluating federally funded research programs.(7)
Peer review is also used for scientific and technical products relating to policies, including regulations, to determine whether the underlying scientific findings are well supported. For example, such peer review was established as Environmental Protection Agency (EPA) policy in 1993.(8) In 1996, a panel of leading economists concluded that peer review should be used for economic analyses supporting regulations with a potentially large impact on the economy (e.g., those whose annual economic costs exceed $1 billion).(9) The panel also indicated that reviewers should be selected based on their demonstrated expertise and reputations.
In its peer review bulletins, OMB recognized the variety of ways that peer review is used. OMB said that independent peer review is especially important in the regulatory arena because federal agencies often develop or fund the science that underlies their regulations, and then oversee the peer review of those scientific studies -- thereby creating the appearance of a conflict of interest.
Legislative Proposals. A number of statutes and legislative proposals have advocated the use of peer review in a regulatory context for particular issues. For example, the Safe Drinking Water Act (42 U.S.C. 300g-1(b)(3)(A)) requires EPA, when taking action under the act based on science, to use the "best available, peer reviewed science and supporting studies." In the 108th Congress, the Water Resources Development Act of 2003 (H.R. 2557) would have generally required project studies to be subject to peer review by an independent panel of experts if the project has an estimated total cost of more than $50 million. Similarly, the Sound Science for Endangered Species Act Planning Act of 2003 (H.R. 1662, 108th Congress) would have directed the Secretary of the Interior, in making decisions about species protection, to give greater weight to certain kinds of data that had been peer reviewed by qualified individuals as defined in the bill.
There have also been legislative efforts to require peer review more broadly. For example, in the 106th Congress, the Senate considered but did not enact bipartisan legislation (S. 746) that would have required virtually all agencies to provide for an independent peer review of any required risk assessments and cost-benefit analyses of major rules that the agencies or the OMB Director reasonably anticipated were likely to have a $500 million effect on the economy. The bill would have required that peer reviews be conducted through panels that were "broadly representative" and involved participants with relevant expertise who were "independent of the agency." However, if an agency certified that adequate peer review had already been conducted, and the OMB Director agreed, no further peer review would have been required. In its comments on this legislation, the General Accounting Office (GAO) (now the Government Accountability Office) generally supported the use of peer review in this context, noting that "the rigorous, independent review of economic analyses should help enhance the quality, credibility, and acceptability of agencies' decisionmaking."(10) However, GAO cautioned that (given the number of reviews contemplated) agencies would need to plan carefully for the reviews, and that the panels would need to reflect all points of view.
Presidential Support. Recent presidential administrations have also supported peer review as a preferred means of assessing scientific research, both prospectively and retrospectively. For example, beginning with the FY1996 budget cycle, OMB and OSTP have jointly provided annual direction to agencies, encouraging them to emphasize the funding of peer-reviewed research over nonpeer-reviewed research for most scientific activities. In 1997, the then-Administrator of OMB's Office of Information and Regulatory Affairs (OIRA) testified that the Clinton Administration supported peer review, but also said the administration recognized that it is not cost-free in terms of agencies' resources or time.(11)
On September 20, 2001, the new OIRA Administrator for the George W. Bush Administration issued a memorandum for the President's Management Council recommending (among other things) that agencies subject regulatory impact analyses and supporting technical documents for "economically significant" and "major" rules (e.g., those with a $100 million annual effect on the economy) to independent, external peer review.(12) The OIRA Administrator also recommended certain criteria for peer review (e.g., disclosure by peer reviewers of prior technical or policy positions on the issues at hand and their sources of personal and institutional funding), and said OIRA would give agency analyses that had undergone such a review "a measure of deference" during its reviews of their regulatory proposals under Executive Order 12866.(13)
Although the concept of peer review has generally received wide support, some observers have also raised cautionary notes or mentioned certain limitations to the approach in particular situations. For example, as OMB indicated in its 1997 testimony, peer review costs can be significant for agencies in terms of both time and agency resources.(14) To pay for peer review procedures, agencies may have to divert resources from other areas (e.g., regulatory enforcement or standards development). Scientists from academic institutions who perform peer reviews on a voluntary basis may also incur opportunity costs with respect to other activities such as teaching and research. Other concerns have focused on how peer reviews have been implemented. For example, GAO noted that peer review methods varied within and among federal agencies,(15) and that agencies' economic analyses of major rules were often not peer reviewed.(16) A number of observers have expressed concerns about the impact of peer review requirements on the pace of regulatory activity, with additional requirements exacerbating what is already often regarded as an "ossified" rulemaking process.(17) In fact, some critics have suggested that regulatory relief and delay is the primary purpose of peer review proposals in this context.
Still other concerns about peer review have centered on issues of bias and balance. Experts agree that effective peer review panels must be (and must be perceived to be) free from any significant conflict of interest and properly balanced, allowing for a wide range of views and appropriate expertise. However, in June 2001, GAO reported that the policies and procedures developed by EPA's Science Advisory Board to ensure balance and independence of the Board's peer reviewers had limitations that reduced their effectiveness.(18) For example, GAO said that the staff office did not routinely ensure that panel members' financial disclosures were complete or that they contained enough information to determine whether a conflict of interest existed. Also, panel members reportedly did not have to disclose information regarding previous positions on the matter being reviewed until the panel's first meeting, thereby making it difficult to determine the independence and balance of the panel members until after they had been selected. GAO also reported that the panel's policies and procedures did not adequately inform the public about the points of view represented on the panels. GAO made several recommendations in this report that were designed to better ensure that the Science Advisory Board's peer review panels are independent and balanced.
Procedural Flexibility or Uniformity. As noted previously, GAO reported that federal agencies' peer review practices are not consistent.(19) However, several studies of peer review in the federal government have suggested that rigidly uniform peer review procedures may not be desirable. For example, a 1996 National Science and Technology Council report indicated that peer review implementation should be flexible and "appropriate to the nature of scientific processes."(20) Similarly, in its 1999 report on peer review, GAO reported that OSTP believed agencies' peer review practices should be "flexible and tailored to agency missions and type of research, and that specific uniform practices should not be dictated for every agency or all federally funded research."(21) GAO also said that agencies viewed a variety of peer review methods as "both appropriate and essential, reflecting the varying nature of the research and its purposes, the differences in research timelines, the broad spectrum of [research and development] performers, and the varying funding mechanisms, such as grants, contracts, and cooperative agreements." Notably, although GAO pointed out that peer review practices varied across and within agencies, it did not recommend greater uniformity in those practices.
OMB said that the final peer review bulletin was being issued "under the Information Quality Act [IQA] and OMB's general authorities to oversee the quality of agency information, analyses, and regulatory actions."(22) The IQA (also known as the "Data Quality Act") was enacted in December 2000 as Section 515 of the Treasury and General Government Appropriations Act for Fiscal Year 2001 (P.L. 106-554). The act supplemented the Paperwork Reduction Act (PRA), which already required OMB to "develop and oversee the implementation of policies, principles, standards, and guidelines to apply to federal agency dissemination of public information."(23) The PRA also required agencies to manage their information resources to "improve the integrity, quality, and utility of information to all users within and outside the agency."(24)
Although little noticed at the time it was enacted, the IQA has subsequently been the subject of significant debate and controversy.(25) The act required OMB to issue guidance to federal agencies designed to ensure the "quality, objectivity, utility, and integrity" of information disseminated to the public. It also required agencies to issue their own information quality guidelines, and to establish administrative mechanisms that allow affected persons to seek correction of information maintained and disseminated by the agencies that does not comply with the OMB guidance. However, it did not require either OMB or the agencies to issue peer review bulletins.
OMB published its final government-wide IQA guidelines in the Federal Register in February 2002.(26) In them, OMB encouraged (but did not require) the use of peer reviews in the development of agency-disseminated information. Specifically, OMB indicated that agencies can presume that data are sufficiently "objective" if they have been subject to an independent peer review process (e.g., as used by scientific journals), but said a member of the public could rebut this presumption "based on a persuasive showing by the petitioner in a particular instance." OMB also indicated that journal peer review may not be sufficient for information likely to have an important public policy or private sector impact. Finally, the IQA guidelines set minimum standards for the transparency of agency-sponsored peer review (referencing the standards in the above-mentioned September 20, 2001, memorandum issued by the OIRA Administrator -- e.g., disclosure of reviewers' prior technical or policy positions and sources of funding).
The final peer review bulletin also cited Executive Order 12866 and OMB's authority to "manage the agencies under the purview of the President's Constitutional authority to supervise the unitary Executive Branch" as other authorities supporting the bulletin's issuance.(27) E.O. 12866 says that OIRA within OMB is the "repository of expertise concerning regulatory issues," and says that agencies should base their decisions on "the best reasonably obtainable scientific, technical, economic, or other information concerning the need for, and consequences of the intended regulation."(28) However, the executive order does not specifically mention peer review.
When it published its proposed peer review bulletin in the Federal Register in September 2003, OMB noted in the preamble to the bulletin the importance of peer review and the sometimes inconsistent nature of its application. For these reasons, the office proposed requiring peer review for "the most important scientific and technical information relevant to regulatory policies that [agencies] disseminate to the public."(29) OMB said the new requirements would supplement, but not supplant, both its own information quality guidelines and any agency-specific peer review requirements for regulatory information. Although OMB noted that its bulletins "are not laws or binding regulations," as a practical matter, OMB's instructions to the agencies carry substantial weight because of its authority to review agencies' budget proposals and most significant regulations before they are published.
In essence, OMB's bulletin proposed that each covered federal agency take three actions: (1) have all "significant regulatory information" that the agencies intend to disseminate peer reviewed; (2) have "especially significant regulatory information" peer reviewed according to even higher standards; and (3) provide OMB with information at least once each year about upcoming scientific studies that might support significant regulatory disseminations, and the agencies' plans for conducting peer reviews of those studies. The proposed bulletin also contained several other, related requirements. The bulletin defined "agency" broadly, including cabinet departments and agencies, independent agencies (e.g., EPA), and independent regulatory agencies (e.g., the Federal Communications Commission and the Nuclear Regulatory Commission).
The proposed bulletin defined "significant regulatory information" as scientific or technical information that qualifies as "influential" under OMB's information quality guidelines and is "relevant to regulatory policies." OMB's information quality guidelines define "influential information" as information that the agency can "reasonably determine" will have a "clear and substantial impact on important public policies or important private sector decisions" when it is disseminated to the public. The proposed peer review bulletin said information is "relevant to regulatory policies" if it "might be used by local, state, regional, federal and/or international regulatory bodies." The bulletin indicated that the peer review for this type of information should be "appropriate" and "scientifically rigorous," but also said the appropriate peer review mechanism would depend on "the novelty and complexity of the science to be reviewed, the benefit and cost implications, and any controversy regarding the science."
For information at the lower end of those dimensions, the proposed bulletin indicated that a review by qualified specialists within a separate part of the agency disseminating the information could satisfy the peer review requirement. Also (as in the information quality guidelines), agencies were allowed to presume that any studies that had already been independently peer reviewed did not have to be reexamined, but the bulletin said this presumption "is rebuttable based on a persuasive showing in a particular instance." Finally, the bulletin said peer review was not needed for significant regulatory information related to national defense or foreign affairs, or information that is disseminated in the course of an individual agency adjudication or proceeding on a permit application.
The proposed bulletin indicated that regulatory information should be considered "especially significant" (and therefore subject to more specific peer review requirements) if (a) the agency intends to disseminate it in support of an economically significant regulatory action (e.g., with a $100 million annual impact on the economy),(30) (b) its dissemination would otherwise have a clear and substantial effect on important public policies or important private sector decisions (with an impact of $100 million in any year), or (c) OMB determines it is of "significant interagency interest" or "is relevant to an Administration policy priority." The bulletin laid out a series of proposed requirements for these types of peer reviews. For example, it said agencies must
Also, agencies were instructed to consult with OMB and OSTP regarding the "sufficiency" of their planned peer review policies and, on request, to discuss with OMB the sufficiency of their plans to review specific documents.
The proposed bulletin indicated that, at least once each year, each agency must provide OMB with a summary description of any "existing, ongoing, or contemplated" studies that might support significant regulatory information that the agency intends to disseminate within the next year. Each agency was also to provide OMB with the agency's plan for conducting peer reviews of such studies under the bulletin's requirements, including an agency contact person. OMB indicated that, to reduce paperwork, this information was to be included in other reports being submitted to OMB under Executive Order 12866 or the Information Quality Act.
The proposed bulletin also contained several other peer review requirements or provisions, including the following:
OMB initially said these peer review requirements would apply to information disseminated on or after January 1, 2004. OMB initially requested public comments on the proposed bulletin by October 28, 2003, but later extended the comment period to December 15, 2003, and then to January 16, 2004.
OMB said that it received 187 comments on the proposed bulletin on peer review, and posted those comments on its website.(32) In addition to this formal commenting process, OMB and OSTP sponsored an open workshop on the peer review requirements in November 2003.(33) The workshop was hosted by the National Academy of Sciences (NAS), which subsequently endorsed the bulletin's objectives but expressed several concerns. OMB said the comments that it received varied substantially, with some highly favorable and others suggesting that the bulletin be withdrawn and reconsidered. In general, supporters of the proposed bulletin reportedly indicated that peer review standards across the government were inconsistent, and that more consistent use of peer review could increase the technical quality and credibility of regulatory science. They also reportedly asserted that peer review could protect science-based regulations from political criticism and litigation. For example, the U.S. Chamber of Commerce said that OMB had "provided a Bulletin of high quality and sound judgment," and strongly supported "OMB's efforts to improve the quality, objectivity, utility, and integrity of information disseminated by the federal government to the public."(34) Similarly, the National Petrochemical & Refiners Association strongly supported OMB's proposed bulletin, saying that it would "correct the current variability in federal agency implementation of peer reviews by providing the needed guidance on process requirements."(35)
OMB said that many critics of the proposed bulletin viewed it as an effort to inject political considerations into the world of science, and to use the uncertainty that inevitably surrounds science as an excuse to delay new rules that could be costly to regulated entities. They also reportedly said that the bulletin appeared more concerned about peer reviewers' possible conflicts of interest with agencies than their potential conflicts with regulated industries.
In its summary of the public comments, OMB identified a number of broad themes and noted how it responded to those comments.(36) Specifically, OMB said the comments centered on such issues as the need for the bulletin, its scope and applicability, the prescriptive nature of the review process for "especially significant" information, and the costs of implementing the bulletin.
OMB said many commenters indicated that it was unclear why the bulletin was needed. For example, the American Public Health Association said it was "unaware of any evidence that the current system is not working or any examples of inappropriate or flawed federal regulations being promulgated as a result of failure to peer review."(37) In response, OMB said it would provide additional information in the revised bulletin on the importance of peer review and the variability of agencies' peer review policies.
A number of commenters reportedly said the scope and applicability of the bulletin was unclear.
Commenters also said that the proposed bulletin was too prescriptive in the procedures applicable to "especially significant" regulatory information, and said extensive peer reviews would be costly and could lead to delays in regulatory decisions. In response, OMB said that it would give agencies additional discretion in the revised bulletin to determine the level of peer review required for any particular document, and to use alternative procedures if they could demonstrate that those procedures would meet the requirements in the government-wide and agency-specific IQA guidelines. The response included significant discussion of the current and potential role of NAS in conducting peer reviews. In its comments, NAS had pointed to the role its committees have played in performing peer reviews of various agency programs, activities, policies, and proposals. In determining the appropriate level of review, OMB said agencies should consider the costs and benefits of peer review. OMB also emphasized that its role is not to veto studies, but rather to ensure that they have the appropriate level of review before implementation.
OMB also said it was incorporating suggestions regarding the selection of peer reviewers, including that agencies solicit reviewers from the public and from scientific and professional societies. One major concern of the commenters was that the criteria outlined in the proposed bulletin (e.g., discouraging participation of reviewers who were seeking or receiving funding via research grants from an agency) could result in the most qualified reviewers being excluded from consideration. OMB also said that the revised bulletin would make clear that the receipt of research grants through peer-reviewed competitions does not disqualify a potential reviewer.
According to OMB, many of the commenters raised concerns about public participation in the peer review process, questioning (among other things) whether that participation would be useful and raising logistical considerations. As a result, OMB said it had decided to allow the agencies to decide whether public participation would be allowed and in what forms. However, OMB said it would provide for public comment on the agencies' peer review plans for upcoming documents.
Finally, OMB said some of the commenters requested that OMB provide an estimate of the costs associated with implementation of the peer review bulletin. For example, in a March 2004 letter to OMB, Senators Joseph Lieberman and Richard Durbin said it was "curious that OIRA -- an office known for insisting on cost-benefit analysis -- provides no estimate of costs, nor documentation of the benefits gained or specific harms that will be reduced by the proposal. We suggest that the proposed rules do not pass OMB's own test of good rulemaking."(40)
OMB estimated that there would be about 125 "influential" scientific documents covered each year that are related to rulemaking, and at least 1,250 other documents each year that are unrelated to rulemaking. OMB said some of these documents may be exempt or otherwise not require peer reviews (e.g., they had already been adequately peer reviewed), but did not quantify that assertion. OMB said that a "high-quality individual letter review" would cost an agency about $5,000 per document. Therefore, assuming that 1,000 of the 1,250 documents required peer review and could be reviewed by letter, OMB said the cumulative cost of these reviews would be $5 million.
OMB said "one or two dozen" of these documents could be considered "highly influential scientific assessments" requiring more intense peer review. OMB estimated that an average peer review of these documents would cost $50,000. Therefore, if 20 of these "highly influential" assessments were peer reviewed, the cost would be $1 million. OMB said these costs are "likely large overestimates of the incremental costs of the Bulletin because many agencies already engage in peer review practices." The response also included an analysis that provided a methodology for agencies to use in deciding when a more expensive peer-review study by NAS would be cost-effective.(41)
Several other types of comments were offered on the proposed guidelines that were not included in OMB's summary. For example:
The comments that OMB received on the proposed bulletin led the office to release what it described as a "substantially revised" bulletin in April 2004. The changes were characterized in the media as a "partial retreat" in OMB's initiative.(47) OMB said the revised bulletin provided a more extensive discussion of why government-wide peer review guidance was needed, gave agencies more discretion in determining the appropriate type of peer review for specific products, and more clearly indicated that the guidance does not create any new rights for litigation. Also, the revised bulletin permitted an agency (under certain circumstances) to use a scientist as a peer reviewer who had received research grants from the agency -- a practice that was discouraged in the proposed bulletin. The table appended to this report provides a side-by-side comparison of the proposed and revised bulletins. The sections below discuss similarities and differences in the documents.
Among the more notable changes made to the bulletin were revisions to the scope and applicability of the peer review requirements. In general, the scope of the revised bulletin was arguably much broader than the proposed bulletin. For example, whereas the proposed bulletin applied to "significant regulatory information," the revised bulletin applied to "influential scientific information" -- a scope that included, but was not limited to, regulatory information.(48) However, the revised bulletin also made it clear that certain types of information within this broad category did not have to be peer reviewed, including:
The revised bulletin also explicitly exempted from its requirements information that (1) was related to national security, foreign affairs, or international trade or treaties where compliance with the bulletin "would interfere with the need for secrecy or promptness";(51) (2) was produced by government-funded scientists that was not represented as the views of the agency; (3) was an agency regulatory impact analysis or regulatory flexibility analysis under Executive Order 12866;(52) or (4) was medical, health, or safety information that was time sensitive or based on a recent clinical trial that had already been adequately peer reviewed. The bulletin also allowed agency heads to waive or defer the bulletin requirements "where warranted by a compelling rationale."(53)
In structure, the revised peer review bulletin was similar to the proposed bulletin in that it still essentially required agencies to take three actions (to the extent permitted by law):
Each of these required actions is described more fully in the sections below.
The revised peer review bulletin defined the term "influential scientific information" as information the agency "reasonably can determine that the dissemination of which will have or does have a clear and substantial impact on important public policies or private sector decisions." Therefore, in contrast to the proposed bulletin, the peer review requirements in the revised bulletin were not limited to regulatory policies.
In addition to the limitations and exemptions discussed above, the revised bulletin said that agencies were also not required to conduct a peer review of influential information that had already had an "adequate" peer review. The bulletin gave agencies substantial discretion in determining whether a prior review was "adequate," specifically stating that the earlier review "need not comply with all of the requirements of this bulletin." It also said that the sufficiency of the prior review depended on the "novelty and complexity of the science to be reviewed and the benefit and cost implications." Language in the proposed bulletin about rebutting an agency's presumption regarding whether earlier peer reviews were sufficient was not included in the revised bulletin.
The revised bulletin (like the proposed bulletin) also gave agencies discretion in determining the appropriate peer review mechanism for any information not previously reviewed, with the level of rigor required again dependent on the "novelty and complexity of the science to be reviewed and the benefit and cost implications." The bulletin stated that "appropriate" reviews could range from examinations by other federal specialists to formal panels composed of independent experts. The preamble to the bulletin indicated that some peer reviews could be done by letter (e.g., when a draft document covers only one discipline), but said panels were preferable when time and resources permitted their use. Peer reviewers were to be selected based on their relevant expertise, and were not to have participated in developing the product being reviewed. The revised bulletin also contained requirements to handle participation by scientists with conflicts of interest,(54) and said that the comments of peer reviewers or a summary of those comments should be made available to the public.
When Peer Review Is Required. In response to concerns that the trigger for peer reviews was too vague, OMB indicated in the revised bulletin that the reviews were needed whenever an agency "reasonably determines" that scientific information "will have or does have a clear and substantial impact on important public policies or private sector decisions."
This change did not appear to address one of the underlying concerns voiced in the comments on the proposed bulletin. The definition of "influential" information is essentially the same as the one used to define "significant" information in the proposed bulletin; the only difference is the broader applicability to non-regulatory information. Therefore, agencies still may find it difficult to determine whether a particular study will, at some point in the future, have a "clear and substantial impact on important public policies," much less "important private sector decisions." Two agencies in analogous situations may reach different conclusions regarding the need for peer review. As a result, the overall purpose of the guidelines -- a more consistent application of peer review among federal agencies -- may not be achieved.
The revised bulletin contained a number of additional peer review requirements for "scientific assessments" that are "highly influential." These requirements were much more specific than those placed on "influential" scientific assessments, differed in some respects from the requirements in the proposed bulletin, and appeared to give OMB and the agencies significant authority to determine when the requirements are applicable.
When Enhanced Peer Reviews Are Required. The revised bulletin defined a "scientific assessment" as "an evaluation of a body of scientific or technical knowledge which typically synthesizes multiple factual inputs, data, models, assumptions and/or applies best professional judgement to bridge uncertainties in the available information." The term "highly influential scientific assessment" was defined in the revised bulletin as "influential" scientific information (as defined in the previous section of this report) that either the disseminating agency or OMB determined
As discussed previously, in the proposed bulletin enhanced peer review requirements applied to "especially significant" influential information expected to be disseminated in support of a regulatory action that could have a $100 million impact in any year. Therefore, although the dollar threshold for when enhanced peer review is required was higher in the revised bulletin ($500 million instead of $100 million), the peer review requirements were more broadly applicable in that they applied to more than just regulations. Also, the language regarding "precedent setting, novel, and complex approaches" replaced criteria in the proposed bulletin in which OMB alone could decide that certain information required enhanced peer review because it was of "significant interagency interest" or "relevant to an Administration policy priority."
Selection of Reviewers. The revised bulletin described in some detail how peer reviewers of highly influential scientific assessments should be selected, but still gave agencies substantial discretion in making the final decision. For example, the bulletin said that reviewers must be selected primarily on the basis of "necessary" expertise, experience, and skills, and must be diverse enough to "fairly" represent different perspectives. The revised bulletin also said that agencies must consider requesting the public to nominate reviewers, and must generally "bar participation by scientists employed by the agency sponsoring the review." (As noted previously, the proposed bulletin said agencies should "strive" to select peer reviewers who were independent of the agency.) In addition, as in the proposed bulletin, the revised bulletin said agencies should try to avoid repeated use of the same reviewer on multiple assignments.
However, in a significant departure from the proposed bulletin, the revised bulletin also said that research grants awarded to a scientist through a competitive, peer-reviewed process did not disqualify that scientist from serving on a peer review panel. (The proposed bulletin indicated that peer reviewers who were "receiving or seeking substantial funding from the agency through a contract or research grant" should not be selected as peer reviewers.) Also, whereas the proposed bulletin advocated balancing panel members' biases, the revised bulletin contained no language to that effect. Finally, the revised bulletin said that agencies could commission entities independent of the agency to select peer reviewers and manage the process.
Public Comments. The revised bulletin also gave agencies discretion in the area of public comments. As noted previously, the proposed peer review bulletin instructed agencies to permit interested agencies and persons to submit comments on the information being reviewed, and to provide those comments to the peer reviewers for their consideration. In contrast, the revised bulletin said that agencies could decide on their own whether to make the information at issue available for public comment, but said agencies should "consider" having a public comment period and public meetings before the reviewers "when there is sufficient public interest." The revised bulletin did not indicate how agencies are to make this determination, or how much public interest is "sufficient" to trigger the need for public comments. The bulletin also said that if an agency decides to make a draft assessment available to the public, it must, "whenever practicable," provide peer reviewers with a summary or compilation of the comments received.
Other Requirements. The revised bulletin contained several other types of requirements that were also in the proposed bulletin. For example, the revised bulletin required agencies to provide reviewers with "sufficient information" to allow them to understand the data, assumptions, and analytic procedures used to support key findings or conclusions. Also, agencies were required to instruct peer reviewers to prepare a report summarizing their findings and conclusions. The report must contain the names, affiliations, and credentials of the reviewers. Agencies were, in turn, required to prepare a written response to the report, and both the report and the response must be posted on the agency's website.
Alternative Procedures. The revised peer review bulletin said that an agency can (1) rely on scientific information produced by NAS, (2) commission NAS to peer review the information, or (3) use an alternative procedure "specifically approved by the [OIRA] Administrator, in consultation with OSTP." As noted previously, the bulletin permitted the agency head to waive or defer some or all of the peer review requirements for "influential scientific assessments" and "highly influential scientific assessments" "where warranted by a compelling rationale."(55) If the requirements are waived, however, the bulletin said peer review should be conducted "as soon as practicable thereafter." The proposed bulletin did not provide for alternative procedures.
OMB Authority. As is apparent from the preceding discussion, the revised peer review bulletin (like the proposed bulletin) gave OMB significant authority to decide when agencies should use the more rigorous form of peer review. Specifically, the revised bulletin indicated that these procedures should be used when OMB (or the agency) determines that a scientific assessment (1) "could have a clear and substantial impact on important public policies ... or private sector decisions with a potential effect of more than $500 million in any year," or (2) involves "precedent setting, novel, and complex approaches, or significant interagency interest." Therefore, under this definition, OMB could unilaterally determine that certain information that an agency intends to disseminate is sufficiently "novel" or "complex" to trigger these "highly influential" additional peer review requirements.(56)
Also, whereas the revised bulletin defined "influential"scientific information as information that the agency can reasonable determine "will have or does have" a clear and substantial impact on important public policies or private sector decisions, the bulletin said that information should be considered "highly influential" if either the agency or OMB determines it "could" have such an impact -- conceptually, a lower threshold than for information that is only "influential."
OMB retained significant authority over the peer review process in other ways. For example, OMB could determine that an agency's alternative peer review procedures are sufficiently rigorous to replace the requirements in the revised bulletin.
The proposed bulletin required agencies to notify OMB at least once each year of any existing or upcoming studies that might trigger the peer review requirements within the next year, and of the agencies' plans for conducting those peer reviews. In contrast, the revised bulletin required each covered agency to post an "agenda" on its website every six months delineating any information disseminations subject to peer review. The revised bulletin was also much more specific about these notices than the proposed bulletin, requiring each entry on the agenda to contain a detailed description of the "peer review plan," including:
Each peer review plan was also required to describe how reviewers will be selected, whether the public will be asked to nominate peer reviewers, whether there will be opportunities for the public to comment on the information being reviewed, and whether the agency will provide reviewers with copies of any public comments before beginning their work. The revised bulletin required agencies to establish a mechanism to allow the public to comment on the adequacy of the peer review plans, and must consider those comments.
OMB said that 57 individuals and organizations commented on the April 2004 revised peer review bulletin.(57) About a dozen of the comments essentially asked OMB to allow more time for public comments and interagency review. Most of the substantive comments provided were supportive of at least some of the changes that OMB made to the bulletin. However, some commenters believed the changes had weakened the bulletin to such an extent that they withdrew their initial support, while others believed the changes had not gone far enough. Some of the individuals' and organizations' comments are summarized below.
The presidents of the National Academy of Sciences and its two affiliated institutions, the National Academy of Engineering and the Institute of Medicine, released a statement on April 15, 2004 that generally praised the revision, arguing that it would "improve the quality of the government's scientific assessments and ... decision-making," and "better accommodate the diverse circumstances of ... federal agencies."(58) The statement suggested, but did not state, that further improvements were nevertheless desirable.(59)
In May 2004, OMB Watch filed a comment on the revised peer review bulletin, stating that although OMB had made "significant revisions" to the bulletin, it nevertheless had "fail[ed] to correct some of the most fundamental complaints.(60) Specifically, OMB Watch said that:
In its conclusion, OMB Watch recommended that OMB withdraw the bulletin and instead convene an interagency committee to investigate peer review techniques and flaws, or make changes in the bulletin to address their concerns.
The U.S. Chamber of Commerce submitted comments on the revised bulletin in May 2004, stating that the revised bulletin has "fundamental problems that must be resolved," and that it could not support the issuance of a final bulletin until those problems are addressed.(61) The Chamber said it had "deep concerns" about (among other things):
In summary, the Chamber said that the changes OMB made in issuing the revised bulletin "are so severe and debilitating as to eliminate the public benefit of having a common, government-wide minimum standard for peer review."
In May 2004, twelve Members of Congress provided OMB with comments on the revised bulletin, stating that the revision did not address previously expressed concerns that the proposal was "unjustified, overly broad, burdensome, and did not appropriately guard against appointment of reviewers with conflicts of interest," or that it provided OMB with "excessive authority over the production and dissemination of government information.(62) The Members said that the need for the bulletin remained unjustified, that the exemptions in the revised bulletin resulted in an unbalanced approach, and that it created "considerable new burdens on agencies." They also said that "safeguards to ensure the integrity of the process for selection of reviewers have not been included in the Bulletin." In particular, they said there was "no requirement that agencies disclose publicly whether any panel members with a conflict were selected." In conclusion, the Members characterized the bulletin as "a wolf in sheep's clothing," and said it was "designed to prevent the dissemination of government information, stifle public debate, and delay legitimate government regulatory action." They also said that the agencies, not OMB, are "the repository of scientific and technical expertise," and said they should be given "full authority to govern their own peer review procedures."
AAAS filed a comment with OMB in June 2004, saying that the "revised version is much improved" but raising several issues.(63) For example, AAAS:
The editor of AAAS's flagship publication, Science, praised OMB for its efforts to address the concerns of the scientific community, saying that most of the concerns regarding the proposed bulletin were answered in the revision, and that OMB "deserves credit for being a good listener."(64)
A number of other scientific societies also commented on the revised peer review bulletin. For example, an official in the Association of American Medical Colleges was quoted in April 2004 as saying that OMB had been "very responsive," and said the revised document "is much, much more flexible."(65) In July 2004 (after the comment period for the revised bulletin closed), a consortium of 13 scientific societies, mostly in the life sciences, released a general statement on the role of scientific peer review in policy development.(66) The statement cited peer reviews as "critically important tools for policy makers" and laid out a list of eight "considerations" the societies believe will help ensure that agency peer reviews are well designed and performed. While several of the points were also made by other organizations, some provided a somewhat different emphasis. For example, the societies:
As noted previously, on December 15, 2004, OMB released its final bulletin on peer review. OMB said that it had made "minor revisions" to the April version of the bulletin that were "responsive to the public's comments."(67) Perhaps most notably, OMB said that the final bulletin:
OMB also reportedly made a number of what appear to be less significant changes to the final bulletin. For example, OMB said that the final version of the bulletin:
OMB's peer review bulletin is likely to have a significant effect on federal rulemaking and other forms of information dissemination and public policy. That effect is likely to be both direct (through agencies' and OMB's enforcement of the bulletin's requirements) and indirect (through references to the bulletin by others). For example, Section 402 of the Specialty Crops Competitiveness Act of 2004 (P.L. 108-465, signed by the President on December 21, 2004) indicated that a required peer review of the procedures and standards governing the consideration of certain import and export requests "shall be consistent with the guidance by the Office of Management and Budget pertaining to peer review and information quality."
Some of the initial issues and concerns raised by commenters on the proposed peer review bulletin were clarified or otherwise addressed in the revised and final versions of the bulletin. Perhaps most notably, the bulletin now makes it clear that scientists are not prohibited from serving as peer reviewers if they receive research grants from the agency based on investigator-initiated, competitive, peer reviewed proposals. The bulletin also provides exemptions from the peer review requirements for time-sensitive medical, public health, and safety information, and other compelling circumstances. However, a number of other issues remain unclear, including (1) the amount of discretion that agencies will actually have in carrying out the bulletin's requirements, (2) the degree to which the bulletin will accomplish the stated goal of making federal peer review practices more consistent, and (3) the effects of the bulletin on agencies and the federal rulemaking process.
The final peer review bulletin appears to give federal agencies substantial discretion in determining whether peer review is required for specific products and, if so, what type of peer review mechanism is appropriate and who should serve as peer reviewers. For example, the bulletin says agencies need not have peer review conducted on influential scientific information that had already been subject to "adequate" peer review. To determine whether a prior review was "adequate," agencies are directed to consider (among other things) whether the science is "novel" or "complex," and whether it is "important" to decision making. (Notably, though, the final bulletin no longer indicates that an "adequate" peer review "need not comply with" all of the requirements of the bulletin.) Also, the final bulletin says that information should be considered "influential scientific information" if the agency "reasonably can determine" that it will have or does have a "clear and substantial" impact on "important" public policies or private sector decisions. The bulletin says that peer reviewers should be selected to provide the necessary expertise, experience, and skills, and the group of reviewers should be "sufficiently" broad and diverse to "fairly" represent the "relevant" scientific perspectives and knowledge.
On the other hand, the final bulletin also gives OMB substantial discretion in certain areas. For example, the bulletin indicates that OMB can require agencies to use the more exacting procedures for "highly influential scientific assessments" if OMB determines the information is a "scientific assessment" that "could" have a substantial impact on public policies or private sector decisions with a "potential impact" of more than $500 million in any year, or is "novel, controversial, or precedent setting," or has "significant" interagency interest. Also, OMB can unilaterally approve agencies' use of alternative peer review procedures.
The amount of discretion that agencies will actually have in carrying out their peer review programs (or, conversely, the amount of control that OMB will retain) will be apparent only through the bulletin's implementation. The amount of agency discretion (or OMB control) could vary substantially from one administration to another.
OMB indicated that stronger peer review policies were needed because of the importance of the issue and, citing a 1999 GAO report, because of the "variability in both the definition and implementation of peer review across agencies." OMB went on to say that, prior to the development of the bulletin, "there were no government-wide standards concerning when peer review is required and, if required, what type of peer review processes are appropriate." Therefore, OMB said that the bulletin "establishes minimum standards for when peer review is required for scientific information, and the types of peer review that should be considered by agencies in different circumstances."
However, the final bulletin may not provide the desired consistency in peer review definition or implementation. As indicated above, the bulletin leaves many key terms undefined or subject to interpretation, and gives the agencies substantial discretion regarding when certain actions should be taken (e.g., when previous peer reviews are "adequate") and which reviewers should be selected (i.e., those with the "necessary" expertise, experience, and skills). To the extent that agencies are allowed to exercise discretion in these areas, consistency may be forfeited. On the other hand, strict enforcement of uniform procedures established by OMB carries with it a different set of concerns about aggregation of power within the Executive Office of the President, and may be resisted by federal agencies. Also, as noted previously, although GAO reported that agencies' peer review practices were inconsistent, it did not recommend greater consistency, and some view variation in those practices as appropriate and desirable.
A number of commenters expressed concerns regarding the effect that adding new peer review requirements would have on what is already viewed by some observers as a lengthy, "ossified" federal rulemaking process. Somewhat related concerns have been voiced regarding the cost of the requirements to federal agencies, with Senators Lieberman and Durbin suggesting that the requirements "do not pass OMB's own [cost-benefit] test of good rulemaking." In response, OMB provided estimates in the preamble to the revised bulletin indicating that it did not believe the costs would be prohibitive to the agencies. As noted previously, though, in 1997, OMB indicated that peer review costs could be significant in terms of both time and agency resources.(68) Little empirical data are currently available regarding the cost of peer reviews, how they affect the federal rulemaking process, or their effect on the quality of the information being reviewed. It is even less clear how the peer review requirements suggested by OMB will affect those factors, and there appears to be no mechanism in place for collecting such data. There are some indications, however, that the requirements could delay regulatory action.(69)
The AEI-Brookings Joint Center for Regulatory Studies recommended that OMB or some other entity build into the peer review program an evaluation to determine its effect on the quality of regulatory analyses. Any such evaluation could, at least conceptually, include an examination of the cost of the new peer review requirements to federal agencies and its effect on the pace of rulemaking. To determine the effect of the peer review bulletin on these or other factors, baseline information regarding the current state of the art would need to be gathered before the bulletin's implementation.
Even if the data indicate that peer review adds time to the early stages of the rulemaking process, that time may be worth the investment if doing so reduces the likelihood of subsequent judicial challenges to the rules. Peer review may also provide agencies with a preview of likely objections to a rule during the notice and comment phase, thereby allowing them to minimize any weaknesses and respond quickly to adverse comments. All of these factors would have to be considered in any evaluation of the effect of peer review on the regulatory process.
Table 1. Comparison of Proposed, Revised, and Final Peer Review Bulletins
| Issue | Proposed Bulletin | Revised Bulletin | Final Bulletin |
| Target of basic level of peer review | "Significant regulatory information," defined as scientific or technical regulatory information that the agency can "reasonably determine" will have, when disseminated, a "clear and substantial impact on important public policies or important private sector decisions." | "Influential scientific information," defined as scientific information (including but not limited to regulatory information) that the agency "reasonably can determine"will have, when disseminated, "a clear and substantial impact on important public policies or private sector decisions." | Same as revised bulletin. |
| Target of more rigorous peer review | "Especially significant regulatory information," defined as significant regulatory information (1) disseminated in support of an economically significant regulation (e.g., $100 million annual impact on the economy); (2) that could, when disseminated, have a "clear and substantial impact on important public policies or important private sector decisions"(those with $100 million impact in any year); or (3) OMB determines is of "significant interagency interest" or "is relevant to an Administration policy priority." | "Highly influential scientific assessments," defined as influential scientific information that either the agency or OMB determines to be "scientific assessments" (e.g., state-of-science reports, meta-analyses, or risk assessments) that (1) could have a "clear and substantial impact on important public policies or private sector decisions with a potential effect of more an $500 million in any year; or (2) involve "precedent setting, novel, and complex approaches, or significant interagency interest." | Essentially the same as revised bulletin. "Highly influential scientific assessments" defined as influential scientific information that the agency or OMB determine (1) could have a "potential impact" (changed from "clear and substantial impact") of more than $500 million in any year or (2) are "novel, controversial [changed from "complex"]or precedent-setting" scientific assessments (changed from "approaches"). |
| Selection of peer reviewers (independence and balance) | Peer reviewers "should" not have participated in the development of the work product. | Same as proposed bulletin. | Reviewers "shall" not have participated in development of work product. |
| Agency "shall strive to appoint experts who ... are independent of the agency, [and] do not possess real or perceived conflicts of interest." One of the factors to consider in determining independence is whether the individual "is currently receiving or seeking substantial funding from the agency through a contract or research grant." | Agency (or entity selecting the reviewers) must "bar participation by scientists employed by the agency sponsoring the review unless the reviewer's service as a peer reviewer defines the government employment (i.e., special government employees)." However, "[r]esearch grants that were awarded to scientists based on investigator-initiated, competitive, peer-reviewed proposals generally do not raise issues as to independence or conflicts." | Essentially the same as revised, but adds exception for the "rare case" where the agency determines that a premier government scientist (1) is employed in a different agency of a Cabinet-level department, (2) is not in a position of management or policy responsibility, and (3) possesses "essential expertise that cannot be obtained elsewhere." | |
| Also consider whether the individual has previously "advocated a position on the specific matter at issue." | No similar provision. | No similar provision. | |
| Also consider whether the individual has conducted multiple peer reviews for the same agency in recent years. | Agencies "should avoid repeated use of the same reviewer on multiple assignments unless his or her participation is essential." | Agencies "shall" avoid repeated use of the same reviewer unless participation is essential "and cannot be obtained elsewhere." | |
| If obtaining the appropriate expertise requires selecting a reviewer with an apparent bias, agencies should ensure that another reviewer with a contrary bias is selected. | No similar provision. | No similar provision. | |
| Agencies should avoid repeated use of the same reviewer on multiple assignments. | |||
| Selection of peer review mechanism (significant or influential information) | Agencies should select an appropriate peer review mechanism (e.g., review within agency or formal review by independent body) based on the novelty and complexity of the science to be reviewed, the benefit and cost implications, and any controversy regarding the science. | Same as proposed bulletin. | Essentially the same criteria, but also (1) the importance of the information to decisionmakers, (2) the extent of prior peer review, and (3) the factors regarding transparency (see below). |
| Public comments | Agencies are required to "provide an opportunity for other interested agencies and persons to submit comments," and must "ensure that such comments are provided to the peer reviewers with ample time for consideration before the peer reviewers conclude their review and prepare their report." | Agencies are allowed (but not required) to obtain comments on draft scientific assessments. If they do so, they must ("whenever practical") provide a summary or compilation of those comments that address "significant scientific and technical issues." When there is "sufficient public interest," agencies must "consider" establishing a public comment period on the peer reviewers' draft report and sponsoring a public meeting. | "Whenever feasible and appropriate," agencies are required to make the draft scientific assessment available to the public for comment when submitted for peer review and to sponsor a public meeting. If they do so, agencies must ("whenever practical") provide peer reviewers with access to the public comments. |
| Alternatives to the bulletin's procedures | No comparable language | Agencies may (1) rely on scientific information produced by NAS, (2) commission NAS to peer review the information, or (3) use an another procedure specifically approved by OMB. | Same as revised bulletin. |
| Transparency requirements | The agency shall direct peer reviewers ("individually or often as a group") to issue a final report "detailing the nature of their review and their findings and conclusions." The report should also disclose "the names, organizational affiliations, and qualifications of all peer reviewers, as well as any current or previous involvement by a peer reviewer with the agency or issue under peer review consideration." The report "should be included in the administrative record for any related rulemakings." | A detailed summary or copy of the reviewers' comments, as a group or individually, shall be made available to the public." Also, the report shall be part of the administrative record for related agency actions "where appropriate." | The agency or entity managing the peer review shall instruct reviewers to prepare a report describing "the nature of their review and their findings and conclusions." The report must (1) either include a verbatim copy of each reviewer's comments (with or without attribution) or represent the views of the group as a whole, and (2) disclose the names of the reviewers and their organizational affiliations. The report must be disseminated on the agency's website, discussed in any related rulemaking, and included in the administrative record. |
| Upcoming peer reviews | Agencies shall provide OMB at least once each year (1) a description of any existing or upcoming studies that might constitute or support significant regulatory information that the agency intends to distribute within the next year, and (2) the agency's plan for conducting a peer review of such studies. | Agencies shall post on their websites, updated at least every six months, an agenda designating all planned and ongoing information disseminations subject to peer review. For each entry, agencies must provide detailed information in a "peer review plan" (e.g., the subject and purpose of the review, how the review will be conducted, the number of reviewers, how they will be selected, whether the public will be allowed to comment, and whether reviewers will be provided with those comments). Agencies are also required to allow the public to comment on the plan, and to consider those comments. | Same as revised, but agencies also required to provide a link from the agenda to each document made public pursuant to the bulletin. Also, agencies' peer review plans should indicate whether the dissemination will be "influential" or "highly influential," the timing of the review, and whether alternative procedures will be employed. |
| Exclusions | Peer reviews are not required for: -- studies that have already had an "adequate" peer review (with review by a scientific journal considered adequate, but rebuttable); -- regulatory information related to national defense or foreign affairs; or -- information disseminated in the course of an adjudication or permit application. |
Peer review is not required for: -- information produced by government-funded scientists if not represented as the views of the agency (with disclaimer); -- information related to national security, foreign affairs, or negotiations involving international trade or treaties; -- information disseminated in the course of an adjudication or permit application, unless the agency determines it is novel or precedent setting; -- medical, health, or safety dissemination that is time sensitive or based on clinical trial data that was previously peer reviewed; -- regulatory impact analyses or flexibility analyses subject to interagency review under EO12866; or -- routine statistical information. |
Same as revised bulletin, except: -- time sensitive "medical" dissemination not excluded (only "health or safety"); -- underlying data and analytical models used in regulatory impact analyses or flexibility analyses are not excluded; and -- an exclusion added for analyses of routine statistical information. |
| Waivers and deferrals | OMB may waive the peer review
requirements if agency makes a
"compelling case" that waiver is
needed because of: -- an emergency, -- an imminent health hazard, -- a homeland security threat, -- "some other compelling rationale" |
The agency head may waive or defer some or all of the peer review requirements for "influential" and "highly influential" information/assessments "where warranted by a compelling rationale." If deferred, the review should be conducted "as soon as practicable thereafter." | Same as revised bulletin. |
| Annual reports | No provision. | No provision. | By December 15, agencies required to provide OMB with a report containing (1) the number of peer reviews conducted during the fiscal year; (2) the number of times alternative procedures were invoked; (3) the number of times waivers or deferrals were invoked; (4) any decisions to use exceptions in reviewer appointments; (5) the number of panels conducted in public and allowed public comments; (6) the number of public comments provided on review plans; and (7) the number of reviewers recommended by professional societies. |
1. (back) Office of Management and Budget, Executive Office of the President, "Proposed Bulletin on Peer Review and Information Quality," 68 Federal Register 54023 (Sept. 15, 2003). This proposed bulletin had been released to the public via OMB's website on Aug. 29, 2003. To view a copy, see http://www.whitehouse.gov/omb/inforeg/peer_review_and_info_quality.pdf.
2. (back) Office of Management and Budget, Executive Office of the President, "Revised Information Quality Bulletin on Peer Review," 69 Federal Register 23230 (Apr. 28, 2004). This revised bulletin had been released to the public via OMB's website on April 15, 2004; see http://www.whitehouse.gov/omb/inforeg/peer_review041404.pdf. In this report, the first draft of the bulletin is referred to as the "proposed bulletin" and the second draft as the "revised bulletin." Unless otherwise specified, each respective reference includes the introductory supplemental information as well as the body of the bulletin per se.
3. (back) Office of Management and Budget, Final Information Quality Bulletin for Peer Review, Dec. 15, 2004, at http://www.whitehouse.gov/omb/inforeg/peer2004/peer_bulletin.pdf.
4. (back) Office of Management and Budget, Final Information Quality Bulletin for Peer Review, 70 Federal Register 2664 (Jan. 14, 2005).
5. (back) While the term "peer review" is not explicitly defined in OMB's bulletins, the supplementary information in the final bulletin describes peer review generally as "one of the important procedures used in science to ensure that the quality of published information meets the standards of the scientific community. It is a form of deliberation involving an exchange of judgments about the appropriateness of methods and the strength of the author's inferences. Peer review occurs when a draft product is reviewed for quality by specialists who were not involved in producing the draft." Arguably, the methods laid out in the bulletin for conducting peer review could also be considered an implicit definition of the concept.
6. (back) Final Information Quality Bulletin for Peer Review, p. 2.
7. (back) The Committee on Science, Engineering, and Public Policy, Evaluating Federal Research Programs: Research and the Government Performance and Results Act (Washington: National Academies Press, Feb. 1999). The Committee is a joint committee of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine.
8. (back) U.S. General Accounting Office, Federal Research: Peer Review Practices at Federal Science Agencies Vary, GAO/RCED-99-99, Mar. 1999.
9. (back) Kenneth J. Arrow, et al, Benefit-Cost Analysis in Environmental, Health, and Safety Regulation: A Statement of Principles (Annapolis, MD: AEI Press, 1996).
10. (back) U.S. General Accounting Office, Regulatory Reform: Comments on S. 746 -- The Regulatory Improvement Act of 1999, GAO/T-GGD/RCED-99-163, Apr. 1999, p. 6.
11. (back) Statement of Sally Katzen, OIRA Administrator, before the Senate Committee on Governmental Affairs, Sept. 12, 1997.
12. (back) For a copy of this
memorandum, see
http://www.whitehouse.gov/omb/inforeg/oira_review-process.html.
13. (back) Executive Order 12866, "Regulatory Planning and Review," 58 Federal Register 51735, Oct. 4, 1993. For a description of OMB's reviews under this executive order, see CRS Report RL32397, Federal Rulemaking: The Role of the Office of Information and Regulatory Affairs, by [author name scrubbed].
14. (back) Statement of Sally Katzen, OIRA Administrator, before the Senate Committee on Governmental Affairs, Sept. 12, 1997.
15. (back) GAO/RCED-99-99. For example, GAO reported that some agencies conducted peer reviews of research proposals primarily by mail, while others generally relied on panels or committees. All agencies used a combination of external and internal reviewers for these reviews, but one (Federal Aviation Administration) relied primarily on agency employees who were not employed in the project but had the required expertise.
16. (back) U.S. General Accounting Office, Regulatory Reform: Agencies Could Improve Development, Documentation, and Clarity of Regulatory Economic Analyses, GAO/RCED-98-142, May 1998. Only 1 of the 20 economic analyses that GAO reviewed had been independently peer reviewed.
17. (back) See, for example, Thomas O. McGarity, "Some Thoughts on 'Deossifying' the Rulemaking Process, Duke Law Journal, vol. 41 (June 1992), pp. 1385-1462; Richard J. Pierce, Jr., "Seven Ways to Deossify Agency Rulemaking," Administrative Law Review, vol. 47 (winter 1995), pp. 59-93; Paul R. Verkuil, "Rulemaking Ossification -- A Modest Proposal," Administrative Law Review, vol. 47 ( summer 1995), pp. 453-459.
18. (back) U.S. General Accounting Office, EPA's Science Advisory Board Panels: Improved Policies and Procedures Needed to Ensure Independence and Balance, GAO-01-536, June 2001.
20. (back) Subcommittee on Research, Committee on Fundamental Science, National Science and Technology Council, Assessing Fundamental Science (Washington: National Science Foundation, July 1996). The National Science and Technology Council was established by executive order in November 1993 to coordinate federal research and development, and to establish clear national goals for science and technology investments. To view a copy of this report, see http://www.nsf.gov/sbe/srs/ostp/assess/start.htm.
21. (back) GAO/RCED-99-99, p. 5.
22. (back) Office of Management and Budget, Final Information Quality Bulletin for Peer Review, op. cit., p. 2666.
23. (back) 44 U.S.C. 3504 (d)(1).
24. (back) 44 U.S.C. 3506(b)(1)(C).
25. (back) For a more complete discussion of the IQA and OMB's information quality guidelines, see CRS Report RL32532, The Information Quality Act: OMB's Guidance and Initial Implementation, by Curtis W. Copeland and Michael Simpson.
26. (back) Office of Management and Budget, "Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies," 67 Federal Register 8452, Feb. 22, 2002. OMB published proposed guidelines in June 2001 (66 Federal Register 34489), and published an earlier version of the final guidelines in September 2001 (66 Federal Register 49718).
27. (back) Office of Management and Budget, Final Information Quality Bulletin for Peer Review, op. cit., p. 7.
28. (back) Executive Order 12866, section 1(b)(7).
29. (back) 68 Federal Register 54026, Sept. 15, 2003.
30. (back) The definition of an "economically significant" regulatory action can be found in section 3(f)(1) of Executive Order 12866.
31. (back) The bulletin cited Byrd v. EPA, 174 F.3d 239 (D.C. Cir. 1999), "holding that peer review panels selected and supervised by outside consultants are not governed by [FACA]." For more information on federal advisory committees and FACA, see CRS Report RL30260(pdf), Federal Advisory Committees: A Primer, by [author name scrubbed]; or CRS Report RL30795(pdf), General Management Laws: A Compendium, coordinated by [author name scrubbed], pp. 35-37.
32. (back) To view the comments provided on the proposed peer review bulletin, see http://www.whitehouse.gov/omb/inforeg/2003iq/iq_list.html.
33. (back) To view a transcript of this conference, see http://www7.nationalacademies.org/stl/Peer_Review_Transcript.pdf.
34. (back) For a copy of the Chamber's comments on the proposed bulletin, see http://www.whitehouse.gov/omb/inforeg/2003iq/170.pdf.
35. (back) For a copy of the association's comments on the proposed bulletin, see http://www.whitehouse.gov/omb/inforeg/2003iq/111.pdf.
36. (back) Office of Management and Budget, "Summary of Public and Agency Comments on Proposed Bulletin on Information Quality and Peer Review, Including Responses by OMB," Apr. 15, 2004. A copy of this summary is available at http://www.whitehouse.gov/omb/inforeg/peer_review_comment.pdf.
37. (back) Letter to Dr. Margo Schwab, Dec. 11, 2003. To view a copy of this letter, see http://www.whitehouse.gov/omb/inforeg/2003iq/86.pdf.
38. (back) Office of Management and Budget, "Summary of Comments."
39. (back) OMB said that guidance for what constitutes a good regulatory impact analysis is in OMB Circular A-4. To view a copy of this circular, see http://www.whitehouse.gov/omb/circulars/a004/a-4.pdf.
40. (back) To view a copy of this letter, see http://govt-aff.senate.gov/index.cfm?FuseAction=PressReleases.Detail&PressRelease_id=694&Affiliation=R. This letter was not part of the 187 comments that OMB listed on the proposed bulletin.
41. (back) OMB estimated that such a study might cost $1 million and take two years to complete.
42. (back) Letter to Dr. Margo Schwab, Dec. 15, 2003. To view a copy of this letter, see http://www.whitehouse.gov/omb/inforeg/2003iq/153.pdf.
43. (back) Letter to Dr. Margo Schwab, Dec. 23, 2004. To view a copy of this letter, see http://www.whitehouse.gov/omb/inforeg/2003iq/186.pdf.
44. (back) Letter to Dr. Margo Schwab, Dec. 12, 2003. To view a copy of this letter, see http://www.whitehouse.gov/omb/inforeg/2003iq/81.pdf.
45. (back) American Association for the Advancement of Science, Council, "On the OMB Proposed Peer Review Bulletin," Mar. 9, 2004, at http://archives.aaas.org/docs/documents.php?doc_id=434.
46. (back) Robert W. Hahn and Robert E. Litan, Comment on Peer Review and Information Quality, Regulatory Analysis 03-11, Dec. 2003. For a copy of this comment, see http://www.aei.brookings.org/admin/authorpdfs/page.php?id=309.
47. (back) Rick Weiss, "OMB Modifies Peer-Review Proposal: Guidelines Partly Retreat from Strict Control of Agencies' Information Process," Washington Post, Apr. 16, 2004, p. A-19.
48. (back) The explanatory section on requirements in the revised document states that the bulletin "addresses peer review of scientific information disseminations that contain findings or conclusions that represent the official position of one or more Departments or agencies of the federal government." However, the term "official position" is not mentioned in the bulletin itself. The revised bulletin does not include the term "regulatory information." The original proposed bulletin defines "study" as "any research report, data, finding, or other analysis." While this definition is somewhat broader than the meaning usually ascribed to the term by scientists, it is narrower than the definition of "scientific information" in the revised bulletin, which includes "any communication or representation of knowledge ... in any medium or form."
49. (back) FOIA is described in General Management Laws: A Compendium, coordinated by [author name scrubbed], pp. 26-29.
50. (back) The disclaimer is required to say "This information is distributed solely for the purpose of pre-dissemination peer review under applicable information quality guidelines. It has not been formally disseminated by [the agency] and should not be construed to represent any agency determination or policy."
51. (back) The proposed bulletin said agencies "need not have peer review conducted on significant regulatory information that relates to national defense or foreign affairs."
52. (back) As noted previously, OMB has said that the data used to generate these analyses are covered by the peer review requirements. Also, the bulletin did not indicate whether regulatory flexibility analyses under the Regulatory Flexibility Act were exempt.
53. (back) The proposed bulletin allowed OMB (not the agencies) to waive many of the bulletin's requirements if an agency "makes a compelling case" on grounds of an emergency, imminent health hazard, homeland security threat, or "some other compelling rationale."
54. (back) Specifically, the revised bulletin says that the agency (or the entity selecting peer reviewers) must (1) ensure that reviewers who are federal employees comply with federal ethics requirements, (2) apply or adapt those requirements to reviewers who are not federal employees, and (3) consider the National Academy of Sciences' conflict of interest policy.
55. (back) The proposed bulletin would have permitted OMB (not the agency heads) to waive the peer review requirements.
56. (back) In the proposed bulletin, OMB had similar authority. The agency could be required to conduct a formal, independent, external peer review if OMB determined that the information was of "significant interagency interest or [was] relevant to an Administration policy priority."
57. (back) For OMB's response to the comments provided on the revised peer review bulletin, see http://www.whitehouse.gov/omb/inforeg/peer2004/peer_response.pdf. For a link to each of the comments, see http://www.whitehouse.gov/omb/inforeg/peer2004/list_peer2004.html.
58. (back) National Academy of Sciences, "Academies' Presidents Comment on OMB Peer Review Guidelines," Press Release, Apr. 15, 2004, at http://www4.nationalacademies.org/news.nsf/isbn/s04152004?OpenDocument.
59. (back) "...the extended comment period will permit further refinements in the proposed guidance..." (ibid.).
60. (back) For a copy of OMB Watch's comments, see http://www.ombwatch.org/info/dataquality/CommentsRevisedPR-OMBW.pdf.
61. (back) For a copy of the
Chamber's comments, see
http://www.uschamber.com/NR/rdonlyres/eznlbdbxz7wylilw7now26n2cqigrnx57hqpavqkxf7xgvfy2foiy5w5bd37suvonnhln7jv7myxop/COMMENTSProposedRevisedBulletinonPeerReviewandInfo.pdf.
62. (back) For a copy of these Members' comments, see http://www.whitehouse.gov/omb/inforeg/peer2004/25.pdf.
63. (back) For a copy of the AAAS comments, see http://www.aaas.org/spp/cstc/docs/5-28-04OMBpeerltr.pdf.
64. (back) Donald Kennedy, "Praise, for a Change," Science, vol. 304 (May 21, 2004), p. 1077.
65. (back) "White House Softens Disputed Peer-Review Plan," Science, vol. 304 (Apr. 23, 2004), p. 496.
66. (back) American College of Preventive Medicine et al., "Scientific Peer Review in Policy Making," Position Statement, July 15, 2004, http://www.aibs.org/position-statements/040715_scientific_peer_.html.
67. (back) The characterizations in this section of OMB's changes in response to public comments are from OMB's response to the comments provided on the revised peer review bulletin, available at http://www.whitehouse.gov/omb/inforeg/peer2004/peer_response.pdf.
68. (back) Statement of Sally Katzen, OIRA Administrator, before the Senate Committee on Governmental Affairs, Sept. 12, 1997.
69. (back) Ben Geman, "White House Peer Review Requirements Could Delay Standards," Greenwire, volume 10, number 9 (Dec. 17, 2004). The article quoted a program manager in the Department of Energy as saying that efficiency standards for residential furnaces and boilers, commercial air conditioners and other equipment were being delayed for two years because of OMB's peer review bulletin.
Return to CONTENTS section of this Long Report.