Notes
CHAPTER 11 Safeguarding the Norms and Values of Science in the Age of Generative AI
Kathleen Hall Jamieson and Marcia K. McNutt
Revolutionary advances in AI have brought us to a transformative moment for science. AI is accelerating scientific discoveries and analyses. At the same time, its tools and processes challenge core norms and values in the conduct of science, including accountability, transparency, replicability, and human responsibility.…
We call upon the scientific community to establish oversight structures capable of responding to the opportunities AI will afford science and to the unanticipated ways in which AI may undermine scientific integrity.
We propose that the National Academies of Sciences, Engineering, and Medicine establish a Strategic Council on the Responsible Use of Artificial Intelligence in Science. The council should coordinate with the scientific community and provide regularly updated guidance on the appropriate uses of AI, especially during this time of rapid change. The council should study, monitor, and address the evolving uses of AI in science; new ethical and societal concerns, including equity; and emerging threats to scientific norms. The council should share its insights across disciplines and develop and refine best practices.
—Blau et al., “Protecting Scientific Integrity in an Age of Generative AI”
This chapter is premised on the fact capsulized in the opening sentence of the NAS-APPC-Sunnylands (hereafter “Sunnylands” or “working group”) working group statement: “Revolutionary advances in AI have brought us to a transformative moment for science.”1 Many of these transformations are explored in Chapter 3 by Jeannette Wing, Chapter 8 by Eric Horvitz and Tom Mitchell, and in the perspectives pieces in Chapter 9 (for biographies, see Appendix 2).
However, as the Sunnylands statement also notes, AI’s “tools and processes challenge core norms and values in the conduct of science, including accountability, transparency, replicability, and human responsibility.”2 Here we explain the need to safeguard the interrelated scientific norms of transparency and accountability (and, with them, replicability) as well as the ethical principles that shape our understanding of scientists’ responsibilities in the face of transformative changes. In the process, we signal the rationale underlying the working group’s twofold call for monitoring and addressing threats to those norms and ethical values. The first urges “Scientists, together with representatives from academia, industry, government, and civil society … [to] continuously monitor and evaluate the impact of AI on the scientific process, and with transparency, adapt strategies as necessary to maintain integrity.”3 The second calls for establishment of a National Academy of Sciences, Engineering and Medicine (NASEM) Strategic Council on the Responsible Use of Artificial Intelligence in Science to “monitor, and address the evolving uses of AI in science; new ethical and societal concerns, including equity; and emerging threats to scientific norms.”
Whether scientists are probing black holes, microbes, or human psychology, the scientific community which they form is bound together by its commitment to a common set of norms, among them one that requires individual and collective human responsibility for engaging in practices that foster accountability and with it the transparency that makes replicability possible. This combination of commitments provides scientists with the wherewithal to engage in the organized skepticism that fosters a hallmark of science, a culture of critique and correction. That culture in turn incentivizes an ongoing updating of what is known through science’s methods. Scientists are united as well by ongoing efforts to ensure not only that these norms and values are honored but also that ways to honor them are refreshed in the face of changing circumstances. The rapidly evolving capacities of AI have created such circumstances.
The Nature, Function, and Importance of Scientific Norms
Whether thought of as aspirations,4 prescriptions telling scientists how they should behave,5 or myths used by scientists to justify resources, enhance survivability, and burnish perceptions of the legitimacy of their work,6 the norms espoused by science, such as accountability, transparency (and with it replicability), a culture of critique and correction,7 and respect for the ethical limits and obligations in the conduct of their work (i.e., respect for persons, beneficence, and justice), are an integral part of scientists’ self-presentation.8 Together, they encourage scientists “to resist contrary impulses.”9 Structures that incentivize transparency and catching and correcting errors and fraud couple with the inherent competition among scientists to sustain the organized skepticism10 that facilitates both discovery and the production and updating of knowledge.11
Because trust in science increases when scientists and the outlets certifying the trustworthiness of their work honor the norms and values of science,12 the scientific community, in the form of universities, journal families, professional organizations, and entities such as the National Academy of Sciences and National Science Foundation, engages in an ongoing examination of ways to increase adherence to them. For example, the salience of the norm of transparency was bolstered when major journals began requiring preregistration of hypotheses and analysis plans, disclosure of conflicts of interest, and depositing of data and codes as conditions of publication. Along the way, signals of trustworthiness such as checklists and badging have been conventionalized as means of communicating that a publication has honored scientific norms.13 Structures that inculcate norms and instill the value of human accountability include the responsible conduct of research (RCR) education and training that the National Institutes of Health requires of its grantees14 and Institutional Review Boards that superintend research involving human subjects in universities.15
Among the norm-related themes interlaced throughout the various AI governance frameworks explored by professor of Public Policy, Governance, and Innovative Technology, and dean of the TUM School of Social Sciences and Technology at the Technical University of Munich, Urs Gasser, in Chapter 5 are those of concern here, including the need to ensure that AI systems are responsive to human needs, ethical, subject to human oversight and accountability, and transparent.
Human Accountability and Responsibility
Concerns Motivating Calls for a Focus on Human Responsibility and Accountability
There is widespread agreement that AI should “augment human intelligence, not replace it,”16 a sentiment sometimes phrased as the desire to see AI function as a copilot not an autopilot.17 There is agreement as well that vigorous human oversight of the development of AI is needed. Consistent with this view, a March 2023 open letter signed by CEO of Tesla Motors Elon Musk, Apple cofounder Steve Wozniak, Skype cofounder Jaan Tallinn, and a number of “well-known AI researchers”18 called for a six-month “pause” in AI development. “AI labs and independent experts should use this pause to jointly develop and implement a set of shared safety protocols for advanced AI design and development that are rigorously audited and overseen by independent outside experts,” the signatories noted.19
The letter asserted that “Contemporary AI systems are now becoming human-competitive at general tasks” and, in language that some saw as overblown,20 cast dire threats to humankind in questions including “Should we let machines flood our information channels with propaganda and untruth? Should we automate away all the jobs, including the fulfilling ones? Should we develop nonhuman minds that might eventually outnumber, outsmart, obsolete and replace us? Should we risk loss of control of our civilization?” “Such decisions,” it concluded, “must not be delegated to unelected tech leaders.”21
After that letter’s call for a pause went unheeded, two months later a number of the same experts issued a twenty-two-word statement that elicited headlines around the globe. “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war,” it said. Among the goals of that statement was creating “common knowledge of the growing number of experts and public figures who also take some of advanced AI’s most severe risks seriously.”22 As in the case of the letter that preceded it, the credentials of the signatories heightened the credibility of the posited risk. “Published by a San Francisco-based non-profit, the Center for AI Safety,” the statement “has been co-signed by figures including Google DeepMind CEO Demis Hassabis and OpenAI CEO Sam Altman, as well as Geoffrey Hinton and Yoshua Bengio—two of the three AI researchers who won the 2018 Turing Award (sometimes referred to as the ‘Nobel Prize of computing’) for their work on AI,” noted The Verge.23
Agreement on the Need for Human Responsibility and Accountability Expressed in Other Frameworks
The need for human responsibility and accountability has been voiced from the beginning of contemporary deliberations about the future of AI. So, for example, “Responsibility” and “Human Control” were among the Asilomar AI Principles promulgated in 2017 to guide the development of AI. In the words of that influential document:
Responsibility: Designers and builders of advanced AI systems are stakeholders in the moral implications of their use, misuse, and actions, with a responsibility and opportunity to shape those implications.…
Human Control: Humans should choose how and whether to delegate decisions to AI systems, to accomplish human-chosen objectives.24
However, efforts to ensure human responsibility and accountability are complicated by the transformations that AI portends, the pace with which its capacities are evolving, and the opacity of its systems. “Human agency and oversight”25 are as a result focal to guidelines such as the 2019 European Union’s High-Level Expert Group on AI’s Ethics Guidelines for Trustworthy Artificial Intelligence. “AI systems should empower human beings, allowing them to make informed decisions and fostering their fundamental rights,” it declares. “At the same time, proper oversight mechanisms need to be ensured, which can be achieved through human-in-the-loop, human-on-the-loop, and human-in-command approaches.”26 The same focus can be found in the document cast as “the world’s first comprehensive AI regulatory framework,”27 the European Union’s AI Act. “As a prerequisite, AI should be a human-centric technology,” that Act states. “It should serve as a tool for people, with the ultimate aim of increasing human well-being.”28 In a similar manner, the November 2023 Bletchley Declaration by the countries attending the AI Safety Summit, a list that includes the United States, declares that “for the good of all, AI should be designed, developed, deployed, and used, in a manner that is safe, in such a way as to be human-centric, trustworthy and responsible.”29
Human Accountability and Responsibility in the Sunnylands Statement
Consistent with these frameworks, the Sunnylands statement calls on relevant communities to focus sustained attention on five principles of human accountability and responsibility for scientific efforts that use AI.30 The first, “transparent disclosure and attribution,” focuses on transparency. The second’s appeal for “verification of AI-generated content and analysis” focuses on accountability. The third, “documentation of AI-generated data,” aims to achieve both. The fourth calls for “a focus on ethics and equity.” The first four define foci for the “continuous monitoring, oversight, and public engagement” called for by the fifth principle.
Transparency and Accountability
Calls for Transparency and Accountability in Other Frameworks
The context dependence of transparency and oversight requirements for AI is specifically recognized in Article 8 of the Council of Europe Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law, which notes that “Each Party shall adopt or maintain measures to ensure that adequate transparency and oversight requirements tailored to the specific contexts and risks are in place in respect of activities within the lifecycle of artificial intelligence systems, including with regard to the identification of content generated by artificial intelligence systems.” Underlying the EU Act for example, are requirements directly applicable to nation states and their citizenries. These include: “AI systems and their decisions should be explained in a manner adapted to the stakeholder concerned. Humans need to be aware that they are interacting with an AI system and must be informed of the system’s capabilities and limitations.”31 Likewise, a specific facet of governmental systems is the focus of the Asilomar AI principle titled “judicial transparency” that specifies that “Any involvement by an autonomous system in judicial decision-making should provide a satisfactory explanation auditable by a competent human authority.”32
Why Transparency and Accountability Matter in Science
In science, the norm of transparency focuses on disclosure of the scientist’s affiliations (e.g., verified through ORCID), contributions, and potential conflicts of interest (currently documented in contributorship and COI attestations signed when submitting work to a journal), and on the accessibility and usability of the means needed to critique, reproduce, and replicate scientific work. By requiring transparent disclosure of an individual’s contributions to the investigation as well as any relationships and interests that might bias the research process or reporting of it, science ties crediting an author to accountability for a specified facet of the work.33 At the same time, ready access to data, methods, and code and the like make possible five activities that ground reliability in science: reproduction, replication, critique, accountability, and correction.
A study cannot be reproduced or replicated, or its errors caught and the scientific record corrected, unless its data and methods are disclosed and available for critique, reproduction, and replication by others. As the 2019 NASEM report Reproducibility and Replicability recognizes, “transparency [which represents the extent to which researchers provide sufficient information to enable others to reproduce the results] is a prerequisite for reproducibility.”34 To assess the validity of a study’s findings, independent researchers reproduce the study. When they obtain “consistent results using the same input data; computational steps, methods, and code; and conditions of analysis,” the results are considered reproducible. “To help ensure the reproducibility of computational results,” NAS’s 2019 Reproducibility and Replicability in Science report states, “researchers should convey clear, specific, and complete information about any computational methods and data products that support their published results in order to enable other researchers to repeat the analysis, unless such information is restricted by nonpublic data policies. That information should include the data, study methods, and computational environment.”35 Without reproducibility, replication in which a researcher “collects new data to arrive at the same scientific findings as a previous study” is impossible.36
Studies that replicate increase the body of scientific knowledge. “If findings are not replicable, then prediction and theory development are stifled,” notes psychologist and founder the Center for Open Science, Brian Nosek. “If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation.”37
In recent decades, journals have reinforced the norm of transparency and fostered critique, reproduction, and replication by requiring preregistration and public access to data, code, and analysis plans.38 By requiring disclosure of exacting detail, they increase the ability of scholars to reproduce and replicate work. So for example, Science requires that authors “indicate whether there was a pre-experimental plan for data handling (such as how to deal with outliers), whether they conducted a sample size estimation to ensure a sufficient signal-to-noise ratio, whether samples were treated randomly, and whether the experimenter was blind to the conduct of the experiment.”39 And in 2024 both the Proceedings of the National Academy of Sciences (PNAS) and Science added reporting requirements pertaining to the nature of survey research samples and the ways in which they are weighted.40
Scientific disciplines reinforce and refine means of honoring the norm of transparency as well. In 2012, for example, the American Political Science Association (APSA) promulgated guidelines stating that researchers were ethically obligated to “facilitate the evaluation of their evidence-based knowledge claims through data access, production transparency, and analytic transparency.”41 In 2022, APSA updated those requirements to more clearly delineate the expected forms of access:
Researchers have an ethical obligation to facilitate the evaluation of their research or empirical results. Researchers should be explicit about the data sources and methods used, including data sampling, weighting, research design, etc. Researchers should reference the data sources used. If the data were generated or collected by the scholar, researchers should provide access to those data or explain why they cannot. Researchers working with commercial data, big data, text or audio data, social media data, biometric data, digital media archives, geo-located data or confidential data sources that cannot be made publicly available in raw form should provide summary statistics of the data at the finest granulation possible, and clear coding and replication documentation. Attempts to allow others to replicate the analysis should be undertaken. Whenever possible, researchers should provide access to the raw data. Researchers should follow scientific standards for making evidence-based knowledge claims by providing a detailed account of how they draw their analytic conclusions from the data.42
The Threat AI Poses to the Norm of Transparency and, with It, to Reproducibility and Replication
However, the opacity of some data-intensive AI applications makes it difficult for scholars relying on them to understand and disclose their data and decision-making processes. “Most data-intensive AI applications are essentially opaque, ‘black-box’ systems, and new systems capabilities are needed for users to be able to understand the decisions made by the algorithms and their potential impacts on individuals and society …,” noted the National Academies of Sciences, Engineering, and Medicine’s 2022 report, Fostering Responsible Computing Research: Foundations and Practices. “Computing research has only begun to address the need for transparency of these systems.”43
This opacity threatens the scientific norms of transparency and accountability by making it challenging to ascertain model accountability and responsibility and in the process to verify the integrity of AI-generated output including images. “I don’t think I will be able to recognize a good AI-generated image anymore …,” noted image detection sleuth Elisabeth Bik on February in 2024, “there’s probably a lot of papers being produced right now that we can no longer recognize as fake.”44 Both transparency and accountability are called into question when AI technology fabricates images45 and data or plagiarizes.46
The problems associated with opacity are compounded by a second phenomenon—generative AI’s ability to create convincing but fabulated “hyperrealistic content.”47 “For any complex computing system, it is hard to know whether a program does what one intends and expects it to do,” noted Barbara Grosz, Higgins Research Professor of Natural Sciences, Harvard SEAS (see Chapter 9) and the lead scholar on the Fostering Responsible Computing Research report during the Sunnylands deliberations. “Our current inabilities to understand why generative models produce the answers they do, and the ‘hallucinations’ for which they are well known, exacerbate the problem of knowing whether the code they produce actually correctly performs the functions a user intends.”
A further psychological factor increases human susceptibility to the pernicious effects of hallucinations. In a phenomenon known as automation bias, humans, in the words of Fostering, tend to defer to “(automated) computing systems, leading to their disregarding potentially countervailing possibilities or evidence or failing to pursue them.”48 The “neutral computational certainty”49 with which these hallucinations invent content, images, analyses, and attributions and other forms of convincing “hyperrealistic content”50 makes computational bias difficult to counteract or blunt.
Recognizing the importance of safeguarding the transparency norm of science and with it replication, reproduction, accountability, critique and correction, the Sunnylands statement calls for: (Principle one) Transparent disclosure and attribution; (Principle two) Verification of AI-generated content; and, analyses, and (Principle three) Documentation of AI-generated data.
Ethics and Equity
Calls for Ethics and Equity in Other Frameworks
The global landscape of AI includes many AI principles initiatives, as Gasser notes (see Chapter 5). Underlying them are understandings forged from the recognition that the scientific pursuit of knowledge must never be undertaken at the expense of human dignity or autonomy. These ethical frameworks include the Rome Call for Ethics promulgated on February 28, 2020 “to promote an ethical approach to artificial intelligence,”51 a call that grounded its commitments in the Universal Declaration of Human Rights, the milestone 1948 statement by the United Nations (General Assembly resolution 217 A) that defined “a common standard of achievements for all peoples and all nations.”52 In 2024, in a ceremony in Hiroshima’s Peace Memorial Park the representatives of eleven world religions including Buddhism, Hinduism, Zoroastrianism, Bahá’í as well as of the Abrahamic faiths added their names to the list of signatories, a list that already included representatives from tech giants such as Microsoft and IBM.53 Signatories to the Rome Call commit to “the development of an artificial intelligence that serves every person and humanity as a whole; that respects the dignity of the human person, so that every individual can benefit from the advances of technology; and that does not have as its sole goal greater profit or the gradual replacement of people in the workplace.”
The same underlying precepts can be found in the 2019 European Union’s High-Level Expert Group on AI’s Ethics Guidelines for Trustworthy Artificial Intelligence. In that document they take the form of a commitment to “Diversity, non-discrimination and fairness” expressed as statements that “Unfair bias must be avoided, as it could have multiple negative implications, from the marginalization of vulnerable groups, to the exacerbation of prejudice and discrimination. Fostering diversity, AI systems should be accessible to all, regardless of any disability, and involve relevant stakeholders throughout their entire life circle.”54
In Response to Scientific Abuses, Key Ethical Principles Were Codified
Important forms of human responsibility and accountability were codified in the Nurenberg Code, the Universal Declaration of Human Rights, and the Belmont Report in response to abuses of science both in Nazi Germany and in the United States. With its focus on “respect for human rights, individual autonomy, and informed consent,”55 the 1947 Nuremberg Code became “part of the infrastructure of the democratic international system that emerged after World War II.”56 In a similar vein, the 1948 Universal Declaration of Human Rights affirms “the inherent dignity and of the equal and inalienable rights of all members of the human family” and offers the reminder that “disregard and contempt for human rights have resulted in barbarous acts which have outraged the conscience of mankind.”57
Among the science-tied abuses in the United States that gave rise to the Belmont Report58 were the US Public Health Service’s Tuskegee Syphilis Experiment, which scholars have characterized as a “forty year deathwatch,”59 and governmental experiments that exposed human subjects to radiation.60 Durable reforms shaped by the Belmont principles include the Institutional Review Boards (IRBs)61 that govern research supported by the US government62 and the responsible conduct of research principles (RCR)63 that are now a taken for granted part of human subjects research in the United States.
Each of the Belmont Report’s basic principles—respect of persons, beneficence, and justice—led to a requirement. “Just as the principle of respect for persons finds expression in the requirements for consent, and the principle of beneficence in risk/benefit assessment, the principle of justice gives rise to moral requirements that there be fair procedures and outcomes in the selection of research subjects,” noted the Report.64 As Alex John London, K&L Gates Professor of Ethics and Computational Technologies at Carnegie Mellon University, argues in Issues in Science and Technology, “given the esteem the Belmont system has earned, it should be no surprise that concerned parties increasingly argue for its extension to AI innovation.”65
Building upon the Principles of the Belmont Report: Ethics and Equity
In his essay in this volume, London extends the Belmont principles to include two others: nonmaleficence, “generally understood as the duty to avoid inflicting harm or imposing burdens on others,” and fairness, “the duty to treat like cases alike, to apply the same rules or to follow the same process for all individuals, regardless of features or characteristics that are not directly related to some morally relevant aspect of the case.”66 The norm of fairness is reflected in the Sunnylands statement’s Principle Four: Focus on Ethics and Equity.
London’s recommendations are responsive to the need highlighted by working group member Jeannette Wing, Executive Vice President for Research and Professor of Computer Science at Columbia University, who argues in Chapter 3 that the Belmont principles of beneficence, justice, and respect for persons “need to be lifted to operate on groups of individuals, not just on individuals.”67 Among the reasons, explains NAS-APPC-Sunnylands group member Mary Gray, Senior Principal Researcher at Microsoft Research: “even rigid conformity to Belmont principles may not ensure the interests of groups said to be represented by AI models.”68 Recognizing “the interdependence and reciprocity of human beings and the moral significance of caring for others as well as ourselves,” Gray argues that “a researcher dedicated to mutuality might convene their project’s multiple stakeholders, who will determine together what exactly are the risks and rewards of the research and how these will be distributed.”69 And in Chapter 7, Shobita Parthasarathy, Professor of Public Policy and Women’s and Gender Studies at the University of Michigan, and Jared Katzman, PhD Student and Researcher at the University of Michigan School of Information, remind us not only of the negative consequences of the biases in AI datasets and of restriction in AI access but also that technical, organization, and legal policy AI Equity solutions exist as do ones that enhance civic capacity.
These understandings form the backdrop for the Sunnylands statement’s arguments that
Scientists and model creators should adhere to ethical guidelines for AI use, particularly in terms of respect for … the detection and mitigation of potential biases in the construction and use of AI systems.70
Scientists, model creators, and policymakers should promote equity in the questions and needs that AI systems are used to address as well as equitable access to AI tools and educational opportunities.71
Group members also stressed that policymakers cannot effectively address issues of equity and justice merely by “identifying statistical biases in datasets, designing systems to be more transparent and explainable in their decision making, and exercising oversight.”72 Rather, the principle of fairness requires that AI initiatives, in the words of Parthasarathy and Katzman, grapple with “the deep-seated social inequalities that shape the landscape of technology development, use, and governance.”73
In the Sunnylands statement, these values are particularized for the scientific community with specific calls that include:
Scientists and model creators should take credible steps to ensure that their uses of AI produce scientifically sound and socially beneficial results while taking appropriate steps to mitigate the risk of harm.… Scientists and model creators should adhere to ethical guidelines for AI use, particularly in terms of respect for clear attribution of observational versus AI-generated sources of data, intellectual property, privacy, disclosure, and consent, as well as the detection and mitigation of potential biases in the construction and use of AI systems.… Scientists, model creators, and policymakers should promote equity in the questions and needs that AI systems are used to address as well as equitable access to AI tools and educational opportunities.74
The Need for Ongoing Monitoring
As the report, Fostering Responsible Computing Research, on which the Sunnylands statement builds, notes, “A plan for ongoing monitoring and reevaluation by those deploying technologies or otherwise responsible for their governance is needed as research insights make their way into deployed systems and expectations and concerns shift over time.”75 Throughout the deliberations that shaped the Sunnylands statement, working group members reiterated this point. As Grosz put it, “Ethics, societal impact, and responsibility need to be addressed throughout the ‘pipeline’ from ideation and design to deployment.”76 Recognizing this need, the Sunnylands statement’s principle five calls for ongoing monitoring, oversight and public engagement by the scientific community and disciplines within it and by a NASEM AI strategic council.
Past successes testify to the value of such ongoing vigilance by the scientific community. The original February 1975 Asilomar convening on recombinant DNA molecules is a case in point. As those drafting the convening’s report noted, research involving such molecules was developing rapidly and being “applied to many different biological problems.” The conferees responded with a document that not only offered principles for addressing “potential risks” but also noted that “the means for assessing and balancing risks with appropriate levels of containment will need to be reexamined from time to time.” Recognizing that “it is impossible to foresee the entire range of all potential experiments and make judgments on them,” they argued that “it is essential to undertake a continuing reassessment of the problems in the light of new scientific knowledge.”77 Fittingly, in 2017 the Asilomar AI Principles statement made a similar recommendation noting that “advanced AI could represent a profound change in the history of life on Earth, and should be planned for and managed with commensurate care and resources.”78 As the three International Genome Summits that are the subject of the Baltimore-Lovell-Badge case study in Chapter 2 attest, statements by the scientific community can play a role in circumscribing and guiding uses of new technologies.
Despite numerous examples of scientists and other stakeholders taking the initiative to curb abuses of new technology by articulating ethical principles, repeating past successes but now with AI could prove to be far more difficult. AI is already well developed as a commercial product by numerous large corporations, which was not the case with genome editing and other developments that presented ethical concerns. It remains to be seen whether market competition can motivate leadership and developers in commercial AI to adopt principles that guard against abuses or whether competition will disincentivize controls and guardrails.
The call for a NASEM strategic council on AI is based on the presupposition that the academies are the logical home for such an effort. Not only are these entities tasked with advising the nation on scientific matters, but they play an important role in safeguarding the norms of science. As the introduction noted, that norm protective function is evident in reports such as On Being a Scientist: A Guide to Responsible Conduct in Research,79 Fostering Integrity in Research,80 Reproducibility and Replicability in Science,81 Fostering Responsible Computing Research: Foundations and Practices,82 and workshops such as On Leading a Lab: Strengthening Scientific Leadership in Responsible Research.
Creating and superintending the strategic council on AI, as the working group urges, would also comport with the core mission of the academies. Founded by legislation passed by Congress and signed into law by President Lincoln in March 1863, NAS’s mission is providing “leadership in science for the nation and the world by: Recognizing and elevating the best science and fostering its broad understanding. Producing and promoting adoption of independent, authoritative, trusted scientific advice for the benefit of society.”83
Unsurprisingly then after reviewing the history and lessons of past efforts by the scientific community, including the recombinant DNA one, in Chapter 4, Marc Aidinoff, Research Associate at the Institute for Advanced Learning, and David Kaiser, Germeshausen Professor of the History of Science and Professor of Physics, Massachusetts Institute of Technology, draw the conclusion that forms the backdrop of the Sunnylands monitoring, oversight, and engagement principle (principle five).” Specialists and nonexpert stakeholders should regularly scrutinize both evolving technologies and the shifting social practices within which they are embedded. Only then can best practices be identified and refined.”84
Consistent with that conclusion, governments around the globe have instituted structures to monitor and oversee the development of generative AI. The Sunnylands statement recognizes that it is both appropriate and necessary that the scientific community and the disciplines within it do the same. Protecting the integrity of science and its norms and values should be a central focus of such efforts.
Notes
Note to epigraph: Wolfgang Blau et al., “Protecting Scientific Integrity in an Age of Generative AI,” PNAS 121, no. 22 (May 21, 2024), https://
1. Blau et al., “Protecting Scientific Integrity in an Age of Generative AI.”
2. Blau et al., “Protecting Scientific Integrity in an Age of Generative AI.”
3. Blau et al., “Protecting Scientific Integrity in an Age of Generative AI.”
4. Melissa Anderson, Brian Martinson, and Raymond De Vries, “Normative Dissonance in Science: Results from a National Survey of US Scientists,” Journal of Empirical Research on Human Research Ethics 2, no. 4 (December 2007), https://
doi .org /10 .1525 /jer .2007 .2 .4 .3. 5. David Hess, Science Studies: An Advanced Introduction (New York: New York University Press, 1997).
6. John Meyer and Brian Rowan, “Institutional Organizations: Formal Structure as Myth and Ceremony,” American Journal of Sociology 83, no. 2 (September 1977), https://
doi .org /10 .1086 /226550. 7. Bruce Alberts, Ralph J. Cicerone, Stephen E. Fienberg, Alexander Kamb, et al., “Self-Correction in Science at Work,” Science 348, no. 6242 (June 26, 2015): 1420–1422, https://
doi .org /10 .1126 /science .aab3847. 8. Yotam Ophir, Dror Walter, Patrick E. Jamieson, and Kathleen Hall Jamieson, “Factors Assessing Science’s Self-Presentation Model and Their Effect on Conservatives’ and Liberals’ Support for Funding Science,” Proceedings of the National Academy of Sciences 120, no. 38 (September 19, 2023), https://
doi .org /10 .1073 /pnas .2213838120. 9. John Ziman, Real Science: What It Is, and What It Means (Cambridge: Cambridge University Press, 2000), https://
doi .org /10 .1017 /CBO9780511541391. 10. Robert K. Merton, The Sociology of Science: Theoretical and Empirical Investigations (Chicago: University of Chicago Press, 1973).
11. Ophir et al., “Factors Assessing Science’s Self-Presentation Model.”
12. Arthur Lupia et al., “Trends in US Public Confidence in Science and Opportunities for Progress,” Proceedings of the National Academy of Sciences 121, no. 11 (2024): e2319488121.
13. Kathleen Hall Jamieson, Marcia McNutt, Veronique Kiermer, and Richard Sever, “Signaling the Trustworthiness of Science,” Proceedings of the National Academy of Sciences 116, no. 39 (2019): 19231–19236.
14. National Institutes of Health Office of Intramural Research, “Responsible Conduct of Research Training,” National Institutes of Health, https://
oir .nih .gov /sourcebook /ethical -conduct /responsible -conduct -research -training; “Update on the Requirement for Instruction in the Responsible Conduct of Research,” National Institutes of Health, November 24, 2009, https:// grants .nih .gov /grants /guide /notice -files /not -od -10 -019 .html. 15. “Institutional Review Boards: Actions Needed to Improve Federal Oversight and Examine Effectiveness,” US Government Accountability Office, January 17, 2023, https://
www .gao .gov /products /gao -23 -104721. 16. David De Cremer and Garry Kasparov, “AI Should Augment Human Intelligence, Not Replace It,” Harvard Business Review, March 18, 2021, https://
hbr .org /2021 /03 /ai -should -augment -human -intelligence -not -replace -it. 17. Charles Chebli, “The Importance of Using AI as a Copilot, Not Autopilot: Human Intelligence Remains Essential to Artificial Intelligence!,” LinkedIn, June 14, 2024, https://
www .linkedin .com /pulse /importance -using -ai -copilot -autopilot -human -remains -essential -chebli -jpuzf#:~:text =The%20Role%20of%20AI%20as,verify%20AI%2Dgenerated%20insights%20independently. 18. James Vincent, “Elon Musk and Top AI Researchers Call for Pause on ‘Giant AI Experiments’,” The Verge, March 29, 2023, https://
www .theverge .com /2023 /3 /29 /23661374 /elon -musk -ai -researchers -pause -research -open -letter. 19. “Pause Giant AI Experiments: An Open Letter,” Future of Life, March 22, 2023, https://
futureoflife .org /open -letter /pause -giant -ai -experiments /. 20. See, for example, “Expert Reaction to a Statement on the Existential Threat of AI Published on the Centre for AI Safety Website,” Science Media Centre, May 30, 2023, https://
www .sciencemediacentre .org /expert -reaction -to -a -statement -on -the -existential -threat -of -ai -published -on -the -centre -for -ai -safety -website /: “Prof Noel Sharkey, Emeritus Professor of Artificial Intelligence and Robotics, University of Sheffield: ‘AI poses many dangers to humanity but there is no existential threat or any evidence for one. The risks are mostly caused by the Natural stupidity of believing the hype.’ ” 21. Vincent, “Elon Musk and Top AI Researchers.”
22. “Statement on AI Risk,” Center for AI Safety, n.d., https://
www .safe .ai /work /statement -on -ai -risk. 23. James Vincent, “Top AI Researchers and CEOs Warn Against ‘Risk of Extinction’ in 22-Word Statement,” The Verge, May 30, 2023, https://
www .theverge .com /2023 /5 /30 /23742005 /ai -risk -warning -22 -word -statement -google -deepmind -openai. 24. “Asilomar AI Principles,” Future of Life, August 11, 2017, https://
futureoflife .org /open -letter /ai -principles /. 25. High-Level Expert Group on AI (AI HLEG), Ethics Guidelines for Trustworthy AI, European Commission (April 8, 2019), 2, https://
www .europarl .europa .eu /cmsdata /196377 /AI%20HLEG _Ethics%20Guidelines%20for%20Trustworthy%20AI .pdf. 26. Ethics Guidelines for Trustworthy AI, European Commission (April 8, 2019), https://
digital -strategy .ec .europa .eu /en /library /ethics -guidelines -trustworthy -ai. 27. Shiona McCallum, Liv McMahon, and Tom Singleton, “MEPs Approve World’s First Comprehensive AI Law,” BBC, March 13, 2024, https://
www .bbc .com /news /technology -68546450. 28. European Parliament, “Regulation of the European Parliament and of the Council: Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts,” 2024, https://
artificialintelligenceact .eu /the -act /. 29. AI Safety Summit, “The Bletchley Declaration by Countries Attending the AI Safety Summit, 1–2 November 2023,” UK Department for Science, Innovation & Technology, November 1, 2023, https://
www .gov .uk /government /publications /ai -safety -summit -2023 -the -bletchley -declaration /the -bletchley -declaration -by -countries -attending -the -ai -safety -summit -1 -2 -november -2023. 30. Blau et al., “Protecting Scientific Integrity in an Age of Generative AI.”
31. Ethics Guidelines for Trustworthy AI, European Commission (April 8, 2019), https://
digital -strategy .ec .europa .eu /en /library /ethics -guidelines -trustworthy -ai. 32. “Asilomar AI Principles.”
33. Marcia McNutt, “Transparency in Authors’ Contributions and Responsibilities to Promote Integrity in Scientific Publication,” Proceedings of the National Academy of Sciences 115, no. 11 (2018): 2557–2560.
34. National Academies of Sciences, Reproducibility and Replicability in Science (Washington DC: National Academies Press, May 7, 2019), https://
www .ncbi .nlm .nih .gov /books /NBK547537 /. 35. National Academies of Sciences, Reproducibility and Replicability in Science.
36. National Academies of Sciences, Reproducibility and Replicability in Science.
37. Brian Nosek, “Replicability, Robustness, and Reproducibility in Psychological Science,” Annual Review of Psychology 73 (January 2022), https://
doi .org /10 .1146 /annurev -psych -020821 -114157. 38. Edward Miguel et al., “Promoting Transparency in Social Science Research,” Science 232, no. 6166 (January 3, 2014), https://
doi .org /10 .1126 /science .1245317; “Nature Journals Announce Two Steps to Improve Transparency,” Nature 555, no. 6 (February 28, 2018), https:// doi .org /10 .1038 /d41586 -018 -02563 -4. 39. Marcia McNutt, “Reproducibility,” Science 343, no. 6168 (January 17, 2014), https://
doi .org /10 .1126 /science .1250475. 40. Kathleen Hall Jamieson, Arthur Lupia, Ashley Amaya, Henry E. Brady, et al., “Protecting the Integrity of Survey Research,” PNAS Nexus 2, no. 3 (March 2023), https://
doi .org /10 .1093 /pnasnexus /pgad049; PNAS Author Center, “Editorial and Journal Policies,” PNAS (n.d.), https:// www .pnas .org /author -center /editorial -and -journal -policies; “Science Journals: Editorial Policies,” Science (n.d.), https:// www .science .org /content /page /science -journals -editorial -policies#top. 41. A Guide to Professional Ethics in Political Science, American Political Science Association (2012), https://
www .apsanet .org /Portals /54 /APSA%20Files /publications /ethicsguideweb .pdf. 42. A Guide to Professional Ethics in Political Science, American Political Science Association (2022), https://
apsanet .org /Portals /54 /diversity%20and%20inclusion%20prgms /Ethics /APSA%20Ethics%20Guide%20 -%20Final%20 -%20February _14 _2022 _Council%20Approved .pdf ?ver =OshhbBcL94mq7VQiYkp9vQ%3D%3D. 43. National Academies of Sciences, Engineering, and Medicine, Fostering Responsible Computing Research: Foundations and Practices (Washington, DC: National Academies Press, 2022), https://
doi .org /10 .17226 /26507. 44. Deborah Balthazar, “Q&A: The Scientific Integrity Sleuth Taking on the Widespread Problem of Research Misconduct,” STAT: In the Lab, February 28, 2024, https://
www -statnews -com .proxy .library .upenn .edu /2024 /02 /28 /elisabeth -bik -scientific -integrity -research -misconduct /. 45. Jinjin Gu, Xinlei Wang, Chenang Li, Junhua Zhao, et al., “AI-Enabled Image Fraud in Scientific Publications,” Patterns 3, no. 7 (July 8, 2022), https://
doi .org /10 .1016 /j .patter .2022 .100511. 46. Faisal Elali and Leena Rachid, “AI-Generated Research Paper Fabrication and Plagiarism in the Scientific Community,” Patterns (March 10, 2023), https://
doi .org /10 .1016 /j .patter .2023 .100706. 47. Claire Leibowicz, “Why Watermarking AI-Generated Content Won’t Guarantee Trust Online,” MIT Technology Review, August 9, 2023, https://
www .technologyreview .com /2023 /08 /09 /1077516 /watermarking -ai -trust -online /. 48. National Academies of Sciences, Engineering, and Medicine, Fostering Responsible Computing Research: Foundations and Practices.
49. Mike Ananny, “To Reckon with Generative AI, Make it a Public Problem,” Issues in Science and Technology 40, no. 2 (2024): 88, https://
doi .org /10 .58875 /EHNY5426. 50. Leibowicz, “Why Watermarking AI-Generated Content Won’t Guarantee Trust Online.”
51. AI Ethics for Peace, RenAIssance Foundation (July 10, 2024), https://
www .romecall .org /ai -ethics -for -peace -hiroshima -july -10th -2024 /. 52. United Nations General Assembly in Paris, Universal Declaration of Human Rights (December 10, 1948), https://
www .un .org /en /about -us /universal -declaration -of -human -rights. 53. AI Ethics for Peace.
54. High-Level Expert Group on AI (AI HLEG), “Ethics Guidelines for Trustworthy AI.”
55. Jonathan Moreno, Ulf Schmidt, and Steve Joffe, “The Nuremberg Code 70 Years Later,” JAMA 318, no. 9 (2017), doi:10.1001/jama.2017.10265.
56. Trials of War Criminals before the Nuremberg Military Tribunals Under Control Council Law, No. 10 (Washington, DC: US GPO, 1949–1953), https://
science .osti .gov / - /media /ber /human -subjects /pdf /about /nuremburg _code .pdf. 57. United Nations General Assembly in Paris, Universal Declaration of Human Rights.
58. National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, The Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of Research, Department of Health, Education, and Welfare (April 18, 1979), https://
www .hhs .gov /ohrp /regulations -and -policy /belmont -report /read -the -belmont -report /index .html. 59. James Jones, “The Tuskegee Syphilis Experiment” in The Oxford Textbook of Clinical Research Ethics (2008): 86–96. “Congress passed the 1974 National Research Act (Pub. L. 93–348). One result was creation of the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. Charged with identifying “basic ethical principles that should underlie the conduct of biomedical and behavioral research involving human subjects” and with developing “guidelines which should be followed to assure that such research is conducted in accordance with those principles,” the Commission generated the Belmont Report, whose precepts were institutionalized in the form of Institutional Review Boards.”
60. National Museum of Nuclear Science and Technology, “Human Radiation Experiments,” Atomic Heritage Foundation, July 11, 2017, https://
ahf .nuclearmuseum .org /ahf /history /human -radiation -experiments /. 61. Kailee Kodama Muscente, “Ethics and the IRB: The History of the Belmont Report,” Columbia University Teachers College, August 3, 2020, https://
www .tc .columbia .edu /institutional -review -board /irb -blog /2020 /the -history -of -the -belmont -report /. 62. 45 CFR 46, US Department of Health and Human Services, https://
www .hhs .gov /ohrp /regulations -and -policy /regulations /45 -cfr -46 /index .html. 63. Logan Watts, Kelsey E. Medeiros, Tyler J. Mulhearn, Logan M. Steele, et al., “Are Ethics Training Programs Improving? A Meta-Analytic Review of Past and Present Ethics Instruction in the Sciences,” Ethics and Behavior 27, no. 5 (2017): 351–384, https://
pubmed .ncbi .nlm .nih .gov /30740008 /. 64. National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, The Belmont Report.
65. Alex John London, “A Justice-Led Approach to AI Innovation,” Issues in Science and Technology, May 21, 2024, https://
issues .org /ai -ethics -framework -justice -london /. 66. See Chapter 6, “Challenges to Evaluating Emerging Technologies and the Need for a Justice-Led Approach to Shaping Innovation.”
68. Mary L. Gray, “A Human Rights Framework for AI Research Worthy of Public Trust,” Issues in Science and Technology, May 21, 2024, https://
issues .org /ai -ethics -research -framework -human -rights -gray /. 69. Gray, “A Human Rights Framework.”
70. Blau et al., “Protecting Scientific Integrity in an Age of Generative AI.”
71. Blau et al., “Protecting Scientific Integrity in an Age of Generative AI.”
72. Shobita Parthasarathy and Jared Katzman, “Bringing Communities In, Achieving AI for All,” Issues in Science and Technology, May 21, 2024, https://
issues .org /artificial -intelligence -social -equity -parthasarathy -katzman /#:~:text =To%20ensure%20that%20artificial%20intelligence,emerging%20technology%20and%20build%20it. 73. See Chapter 7, “Bringing Power In: Rethinking Equity Solutions for AI.”
74. Blau et al., “Protecting Scientific Integrity in an Age of Generative AI.”
75. National Academies of Sciences, Engineering, and Medicine, Fostering Responsible Computing Research.
76. Barbara Grosz, email message to editors, December 14, 2023.
77. Paul Berg, “Summary Statement of the Asilomar Conference on Recombinant DNA Molecules,” Proceedings of the National Academy of Science 72, no. 6 (June 1975), https://
doi .org /10 .1073 /pnas .72 .6 .1981. 78. “Asilomar AI Principles.”
79. National Academy of Sciences; National Academy of Engineering; Institute of Medicine; Committee on Science, Engineering, and Public Policy, On Being a Scientist: A Guide to Responsible Conduct in Research, 3rd ed. (Washington, DC: National Academy of Science, 2009), https://
nap .nationalacademies .org /catalog /12192 /on -being -a -scientist -a -guide -to -responsible -conduct -in. 80. National Academies of Sciences, Engineering, and Medicine, Fostering Integrity in Research (Washington, DC: National Academies Press, 2017), https://
doi .org /10 .17226 /21896. 81. National Academies of Sciences, Reproducibility and Replicability in Science.
82. National Academies of Sciences, Engineering, and Medicine, Fostering Responsible Computing Research.
83. “About the NAS,” National Academy of Science, n.d., https://
www .nasonline .org /about -the -nas /. 84. Marc Aidinoff and David Kaiser, “Novel Technologies and the Choices We Make: Historical Precedents for Managing Artificial Intelligence,” Issues in Science and Technology, May 21, 2024, https://
issues .org /ai -governance -history -aidinoff -kaiser /.