Policy ForumSCIENTIFIC INTEGRITY

Whose science? A new era in regulatory “science wars”

See allHide authors and affiliations

Science  09 Nov 2018:
Vol. 362, Issue 6415, pp. 636-639
DOI: 10.1126/science.aau3205

The Environmental Protection Agency helped to bring more science into regulation. Polluted air was a major challenge, as seen in Birmingham, AL, USA, 1972.

PHOTO: ENVIRONMENTAL PROTECTION AGENCY

Good laws need good science; however, good science is never guaranteed. Debate over the use of science in law is nearly as old as the laws themselves. With “science wars” waging in health and environmental regulation for at least three-quarters of a century, it is tempting to conclude that recent proposals for reforming regulatory science are similar to what has occurred in the past. They are not. They mark a sharp departure with the past because they legally constrain how agency scientists conduct the initial literature review and synthesis informing policy. Because the reforms generally take the form of legislation or regulation, they do not simply suggest best practices for conducting scientific analyses but establish legal lines that cannot be crossed. Moreover, even though they create legal ground rules for scientific deliberations, the reforms have not been developed by the scientific community, but by members of Congress and political officials. In providing a birds'-eye view of the legal developments in regulatory science over the past 50 years, we identify just how idiosyncratic these current reforms are and why the scientific community needs to be aware of their implications.

Although the agency's underlying scientific analysis is often subject to scrutiny by stakeholders and political officials and review by the courts, these new proposals cut deeper and dictate in part how the formative scientific assessments themselves must be done (1, 2). For example, these proposals require the exclusion of potentially relevant research during agencies' initial review of the literature, dictate the types of computational models that must be considered in analyzing that information, and exclude respected scientists from peer reviewing the analysis (1, 2). If the agency does not respect these legal lines, the agency's review of the scientific literature is legally invalid and technically illegal. This contrasts with present practice where norms governing scientific analyses are rebuttable and subject to modification in light of specific contexts and scientific progress. The proposals thus reach down to control and limit the scientific record.

The scientific community has been vocal in pointing out how the rules diverge from normal scientific practices, even while the legal requirements—some of which are still proposed and others which are final—purport to advance common goals, like data transparency and reproducibility (3). Editors of several journals (including Science), for example, recently observed how one of these proposals conflicts with current scientific norms and practices: “It does not strengthen policies based on scientific evidence to limit the scientific evidence that can inform them; rather, it is paramount that the full suite of relevant science vetted through peer review, which includes ever more rigorous features, inform the landscape of decision making” (3). Scientific analyses subject to these legally prescribed rules are thus at risk of being cordoned off from advancements within the global scientific community.

Science in Regulation

Science in regulation is a distinct scientific practice (4), in which scientists discursively interact and collaborate with lawyers, politicians, and regulatory agencies as part of a process to inform policy decisions. As public decision-makers, these parties also must be accountable. But the scientific aspects of this are scientific practices all the same.

Regulatory science generally follows a two-step process, although the steps are not always explicit (5). First, scientific staff review the available literature and provide a description of what it brings to the policy question at hand. Second, policy-makers can accept, ignore, rerun some of the analysis, or reinterpret the results. This bifurcation of the decision process, which produces a rigorous synthesis and analysis of the literature by scientists before the policy-makers take over, has been endorsed by the U.S. National Academies as the ideal way to make regulatory policy (6, p. 148).

Both steps involve judgments. Even at the first step, substantial discretion is embedded in the scientific review of the available literature. Agency experts have been called on to provide not just point estimates from their syntheses of the literature, but also explanations and descriptions about uncertainties, assumptions, and sources of judgment embedded in the analysis (6). The resulting regulatory science is not perfect, but when done well it signals to policy-makers where there is convergence in the available literature and the nature of the remaining uncertainties (see sidebar). It is also important that agencies' analytical methods keep up with scientific advancements, such as in computational power, or methods such as Bayesian modeling and expert elicitation.

History

The historic arc of regulatory science reveals important innovations and steady progress in providing means of holding agencies accountable for their scientific analyses (7). But this history also spotlights how the most recent proposals mark a departure from the reforms of the past. Rather than allowing a scientific record to be developed and then subjecting that analysis and research to scrutiny and adversarial debate, these current reforms seek to alter the agency's initial analysis of the scientific literature.

Early 1970s: Creation of science bureaucracies

The U.S. government has deployed science in the public interest since its earliest times, for example, with the creation of the patent office in 1802. Federal, science-based environmental laws were passed later, starting in the mid–20th century. Yet they were not well designed from a scientific standpoint. For example, only minimal data collection and analyses were conducted in support of federal water quality requirements. The early 1970s saw the creation of several agencies, like the Environmental Protection Agency (EPA) and other “science bureaucracies,” to address serious environmental and public health problems (4).

Bipartisan legislation passed in the 1970s empowered these expert agencies to carry out research and set standards to advance public health and environmental protection. In these laws, Congress required agencies to base their decisions on science but provided little detail on what that meant. Scientific experts were thus entrusted with both diagnosing and solving society's environmental problems. At the same time, it quickly became clear that agencies needed to be held accountable. Agency scientists could not operate in secrecy.

Almost immediately after agencies began promulgating rules, stakeholders challenged them in court, arguing that certain rules were “arbitrary and capricious” on scientific grounds. Courts sometimes agreed and sent these defective rules back to the agency to justify its decision with evidence from its administrative record. Although a court would not rule on the science, it would insist on “the disclosure of the basis of the agency's action” so that it could determine the legal validity of the decision (8).

Mid-1970s to early 1990s: Emerging importance of accountability

These court decisions highlighted that agency decisions needed to be underpinned by visible scientific explanations. Regulatory agencies devoted more attention to working with the broader scientific community to ensure the scientific integrity of their work. This included the creation of the EPA's Science Advisory Board in 1974, composed of outside experts who review the agency's technical analyses (4). Agencies also relied on external peer review to ensure their decisions were consistent with the scientific evidence and were not unduly influenced by affected parties (4). The changes occurred against a background in which decisions of the EPA were becoming increasingly politically controversial.

Court challenges continued against EPA and other agencies' rules. These cases reinforced the imperative that the agencies not only provide a scientific basis for their regulatory action, but also explain how that evidentiary basis was used to make decisions. For example, if the agency's analyses revealed a range of exposures that might produce health hazards from a pollutant, the agency was expected to explain, with scientific support, why it chose an exposure standard outside the range (7).

Risk assessment was developed during this time as part of a way of structuring the overall regulatory process (6). The important point about the developments of this period is that they were not directly defining science per se. They were regulating the administrative processes to make them more accountable in legal and administrative terms. Judicial pressure impelled the EPA to develop more rigorous analytical processes in response to consistent legal challenges (7).

Early 1990s to mid-2010s: Increased opportunities to challenge the agency's scientific record and scientific analyses

Although court challenges allowed stakeholders to hold agencies accountable for their use of science, they did not allow stakeholders to challenge the scientific record itself as it was being developed. Reforms during this time period opened up opportunities for stakeholders to challenge the rigor of this underlying science, although generally not in ways that were enforced by courts. The Information Quality Act, for example, was passed in 2001 to afford aggrieved parties the right to challenge the reliability of information used by an agency at any point, regardless of whether that information was being used to inform a rule or policy. In another piece of legislation, Alabama Senator Richard Shelby inserted a single sentence into a 4000-page budget bill requiring federally funded researchers to provide their data to anyone who requested them under the Freedom of Information Act, explicitly targeting data underlying a controversial but well-regarded (and reviewed) epidemiological study of health impacts of fine particulate air pollutants (the Harvard Six Cities study) (9).

Political meddling with the agency's scientific record had been a concern in the previous time periods, but during this period, there were more frequent news reports of partisan intervention with agency scientists' own underlying analyses. For example, Julie A. MacDonald, former deputy assistant secretary at the U.S. Department of the Interior, was investigated for unduly influencing field biologists' assessments of the research informing the listing of endangered species and ultimately resigned (10). The White House, under several presidents, was caught altering or censoring staff technical reports in ends-oriented ways (11). The ensuing bad publicity underscored that even though it sometimes occurred, political manipulation of the agency's scientific record was considered off-limits (11).

Evolution of regulatory science behind National Ambient Air Quality Standards (NAAQS)

For 50 years, EPA, working with the scientific community, has developed increasingly rigorous methods for synthesizing the literature to inform the agency's mandate to set ambient air quality standards to protect public health. Part of this evolution is credited to judicial oversight of the administrative process. When EPA failed to explain its scientific analyses or conclusions, courts overturned its standards. We summarize below the evolution of regulatory science of the NAAQS process for particulate matter (PM) using the four stages described in this paper.

Early 1970s

Creation of science bureaucracies. An ad hoc committee of experts from universities, industry, and government synthesized laboratory and epidemiological studies (primarily from New York and London data) and recommended ambient air quality standards for the nation.

Mid-1970s to early 1990s

Emerging importance of accountability. EPA staff took on the role of conducting scientific analyses, which political appointees then used to make decisions about appropriate standards. To ensure the rigor of the staff analysis, EPA and ultimately Congress created a formal, long-term science advisory board to peer review the staff's work [the Clean Air Scientific Advisory Committee (CASAC)]. EPA, working iteratively with the CASAC, collated literature on air pollution and summarized the evidence.

Early 1990s to mid-2010s

Increased opportunities to challenge the agency's scientific record and scientific analyses. Mounting litigation prompted the EPA to streamline and enhance the rigor and transparency of its scientific assessment process. Pursuant to a restructuring of the process in 2006, the EPA now holds a planning workshop for interested parties, conducts a scientific literature search, and prepares risk and exposure assessments with multiple scenarios. The EPA evaluates the strength of evidence of various studies and summarizes the distribution of health effects. The staff also summarizes the results of these technical assessments for nonscientists. All of this work is captured in separate reports, each of which undergoes external scientific and public review. Each of these staff assessments is also firewalled from political control and communication.

Mid-2010s to the present

Efforts to control how scientific analysis and peer review are conducted. If the proposed legal reforms discussed here are finalized, EPA's analyses would be legally altered at the literature review and synthesis stage by rules that proscribe exclusionary tests and other mandated practices that affect how scientists identify and synthesize the available literature. EPA is already making important changes to the composition and other features of external peer review by the CASAC.

Overall, this time period signaled a movement toward greater adversarial pressures on the scientific analysis used for regulation. Additional legal tools made the agency's scientific record more vulnerable to challenge, but it is important to note that these tools did not alter the initial scientific record itself. Scientists would still use their professional standards and methods to determine how to conduct literature searches and analyze the available literature.

During this period, some regulatory programs even required that the agency's initial scientific review and synthesis be firewalled from policy staff and political officials to maintain a strict separation, in keeping with the National Academies' recommendation (5), (6, p. 148). These developments reinforced the importance of scientific integrity. To ensure that no inappropriate pressure was brought to bear on a decision, there was a need to define the scientific basis of decision-making in clear terms. For example, this greater focus on scientific integrity is exemplified by the program to establish National Ambient Air Quality Standards (see sidebar). Additionally, one of President Obama's first actions when he took office in 2009 was to establish a scientific integrity initiative that sought to create stronger protections for the independence of agency scientists.

Mid-2010s to the present: Efforts to control how scientific analysis and peer review are conducted

Presently, the proposed reforms of regulatory science aim to change the nature of the scientific deliberations and underlying record itself. They target the agency's initial scientific analysis and synthesis and prescribe substantial constraints on how this literature review and synthesis must be done. They also alter the composition of the science advisory boards that review these staff analyses. Yet, despite reaching deep down into how scientists assess the available literature, these proposed reforms do not emerge from the scientific community. They are proposed by congressmen and political appointees in the agencies and crafted largely without input or advice of science advisers and mainstream scientific organizations (3).

For example, proposed reforms in Congress and the agencies prohibit agency scientists from including studies in their synthesis of the literature unless the “dose response data and models…are publicly available in a manner sufficient for independent validation” (1). Under the EPA's proposed version of this directive, any exceptions to this transparency requirement must be made by the EPA administrator. Exceptions are narrow and expressly limited to when the disclosure of data is infeasible because of “privacy, confidentiality, confidential business information, and… national and homeland security” (1).

H.R. 1430, legislation that passed the House without amendment in 2017, would establish even more far-reaching restrictions on the nature of information that the agency can consider. Under the terms of the bill, at least the following items underlying “all scientific and technical information” must be “publicly available online in a manner that is sufficient for independent analysis and substantial reproduction of research results” before that information can be used in the agency's scientific analysis supporting a decision: “(i) materials, data, and associated protocols necessary to understand, assess, and extend conclusions; (ii) computer codes and models involved in the creation and analysis of such information; (iii) recorded factual materials; and (iv) detailed descriptions of how to access and use such information.”

These proposed legal standards would apply to all research used to inform a regulatory decision. They even extend to research conducted before the standards were put in place, apparently irrespective of whether compliance with the standards is feasible or even technically possible. The standards if passed as laws would inevitably be enforced by the courts. Lawyers and judges would use these legislated standards to determine what information is needed to allow a study to be capable of validation or replication. As laws, moreover, these legal pronouncements governing scientific deliberations would remain binding and enforceable until they are amended by Congress or formally revised by the appropriate agency.

Changes are also being made to how agency analyses are peer reviewed. Historically, when agencies establish peer-review panels, the composition of reviewers remained flexible and endeavored to enlist the nation's top experts. At the same time, agencies required reviewers to disclose potential conflicts of interest to advance the goals of transparency and balance (4).

By contrast, a 2017 EPA directive by Administrator Pruitt (2) and H.R. 1431, a second bill passed by the House without amendment in 2017, both impose a flat prohibition on the nature of the experts who can serve as peer reviewers of the EPA's analyses. EPA's directive, for example, decrees that “no member of a federal advisory committee [may] currently receive EPA grants, either as principal investigator or co-investigator” (2). In this rule, industry experts with a stake in the proceeding are not excluded from serving on matters in which they or their employers have a financial interest; only researchers holding EPA grants (regardless of the size of the grant) are excluded (2). To our knowledge, there is no precedent for such a unilateral exclusion of federal grantees as peer reviewers in either existing regulatory practice or in the practices of scientific publishers or federal granting agencies. Since EPA issued the directive, at least a few respected scientists have been removed from EPA science advisory boards because they were not willing to abandon their EPA-funded research.

The impacts of the proposed reforms on scientific analysis and the newly enacted rules on peer review are not trivial. For example, if in the wake of EPA's proposed transparency rule, EPA considers a study in its analysis for reasons the scientific community might generally view as meritorious, but the data are not available to the satisfaction of the law, the scientific analysis cannot be used unless the administrator explicitly exempts it (1). And if the top researcher in the country is tapped to help review a staff analysis, but that researcher has an EPA grant, he or she is legally prohibited from doing so.

These initiatives would, in a legally enforceable manner, constrain agencies in determining the best science to fulfill their statutory mandates. They would also limit the ability of scientific staff to use scientific judgment in individual cases; to adopt science innovations that conflict with these proposed legal rules; or to work more generally with the global community of scientists. As such, the proposed legal rules would substantially alter the terms of the open-ended scientific deliberations running through the history of regulatory science. It is important to remember that throughout this history, the agencies were required to explain and disclose the scientific basis of their action as a matter of law.

Moreover, the new reforms introduce legalized requirements for scientific terms, like transparency and reproducibility, that currently are at best in flux and at worst still largely undefined within the scientific community (3). Even proposals within science that sound similar to the proposed reforms are actually very different. In 2014 for example, a committee of academicians, scientific publishers, and funding agencies developed principles to encourage sharing of data, code, research methods and materials, and replication of studies (12). But these principles apply only to encourage better practices for research—they do not suggest that synthesis of the literature, including past studies, should be limited to research that meets these criteria (13). Moreover, even in the research context, this committee recognized that these principles were not universally applicable to all science investigations and described them as aspirational, providing flexibility on how they would be implemented.

Communal Product

As of late September, EPA's proposed transparency rule garnered almost 600,000 comments on the agency's docket, as well as numerous commentaries in leading journals. This substantial feedback may explain in part why the agency recently announced that it will need more than a year to finalize the rule. Some commentators celebrate EPA's proposal, as well as the other reforms discussed here, as a move to replace EPA's open-ended scientific deliberations with legally mandated reproducible science standards. Others warn the proposal will “prohibit the agency from using a wide swath of high-quality, past and present scientific research” (14).

While debates rage about whether the proposed statutes and regulations for science reform will ultimately improve or diminish the quality of scientific evidence, what is clear is that in setting legal standards for what the scientific basis for decision-making is, these proposals mark a clear departure from the past. Scientists not only need to take notice of that fact, but they need to be part of the debate.

Regulatory science cannot and should not be isolated from policy, but science should be allowed to bring its best work to the table. We can all agree with former EPA Administrator Pruitt's statement that “[w]hatever science comes out of EPA, shouldn't be political science” (15). The issue before us is whether this goal will be compromised if legal reform reaches into the earliest stages of the agency's scientific synthesis to narrow the ground rules for these deliberations. Although defining good science has been assigned largely to agency experts, they did not do their work in isolation. It was not only “their” science. The agencies' work benefited from, and was peer reviewed by, the global science community. The agencies' analysis was also expected to keep pace with scientific progress. Finally, agency experts were held accountable to the courts for the choices they made. In the end, the agency's formative scientific analysis was essentially a communal product of science that attempted to summarize what the available scientific information suggests for pressing policy questions of the day. To ignore attempts by politically elected and appointed individuals to dictate how science should be conducted is to betray the very essence of science.

References and Notes

  1. Kennecott Copper Corp. v. Environmental Protection Agency 462 F.2d 846, 849 (DC Cir. 1972).
Acknowledgments: We thank anonymous reviewers and participants at Columbia University's Interdisciplinary Conference on Evidence held in April 2017 for valuable feedback and comments. P. P. was a former employee at the U.S. Environmental Protection Agency. W.W. and E. F. declare no competing interests.

Subjects

Navigate This Article