Contents

• Introduction
• AI Evolution Broadly
• Overview of AI in Health Care
• Techniques and Applications of AI Relevant to Health Care
• AI Applications in Health Care
• Selected Federal Actions
• Executive Order (E.O.) 14110
• Agency Actions
• Assistant Secretary for Technology Policy/Office of the National Coordinator for Health Information Technology
• U.S. Food and Drug Administration
• Office for Civil Rights
• Centers for Medicare & Medicaid Services
• Selected Issues for Consideration
• Conclusion

Summary

The use of artificial intelligence (AI) in health care settings has grown over time in part due to recent increases in health care data availability and innovations in big data analytical methods. AI encompasses multiple technologies and is variably defined, with some disagreement at times among stakeholders about which technologies even qualify as AI. Machine learning (ML) techniques, including deep learning and neural networks, underpin many AI tools used in health care. Some of the more widely used AI techniques and applications in health care include natural language processing, which refers to the use of rule-based or ML approaches to understanding the structure and meaning of human language; rule-based expert systems; physical robots; and robotic process automation. These technologies may affect a wide range of health care stakeholders, including AI developers, researchers, health systems, payers, patients, and federal agencies.

AI technologies have generally become more complex over time, progressing from rule-based expert systems to generative AI models. Within health care, the use of AI broadly falls within three categories: diagnosis and treatment, patient engagement and adherence with treatment plans, and administrative applications. While AI technologies have the potential to improve health care, they may also introduce novel challenges and exacerbate existing ones if not properly overseen.

There have been a variety of federal efforts undertaken to address the use of AI in health care, including by executive orders and agency actions. Agencies within the U.S. Department of Health and Human Services (HHS) that have been active in developing relevant regulations include the Assistant Secretary for Technology Policy/Office of the National Coordinator for Health Information Technology (ASTP/ONC), the U.S. Food and Drug Administration (FDA), the Office for Civil Rights (OCR), and the Centers for Medicare & Medicaid Services (CMS).

These regulatory efforts in part attempt to address challenges that may arise when AI is used in health care settings, including issues related to trust in AI technology, data access, bias, lack of transparency, privacy, scaling and integration, liability, regulatory harmonization, and environmental impact. AI is a rapidly developing and changing technology; if AI is to be used safely and effectively in health care settings, ongoing oversight and regulatory efforts may need to be addressed by Congress and relevant federal regulatory bodies. This report highlights the challenges and questions that Congress may wish to consider.

Introduction

The application of artificial intelligence (AI) in health care settings has become more pervasive due to recent increases in the availability of health care data and innovations in big data analytical methods. When properly trained on relevant clinical data, AI technologies may identify pertinent information from voluminous data, potentially allowing for the rapid improvement of clinical decisionmaking. Certain AI techniques and technologies may also have the ability to "learn" and self-correct using new data, ultimately improving algorithmic accuracy in response to feedback. The use of properly trained AI tools in health care could provide many benefits, including reducing medical errors, improving diagnostics, and streamlining administrative functions. For these benefits to be realized, AI technologies must be trained on data representative of the populations and tasks for which the AI tool is intended. Training data may derive from a variety of clinical practices, including screenings, diagnoses, and treatment plans. Types of training data in these settings include patient demographics, medical notes, electronic health records (EHRs), medical device readings, physical examinations, clinical labs, and images, among others.¹ As these AI technologies and their applications in health care proliferate, stakeholders have begun to pursue regulatory efforts to ensure these technologies are appropriately integrated into the health care setting.

Though AI is often discussed as if it were a single technology, AI is actually an umbrella term encompassing a variety of technologies and techniques. AI has been defined in various ways, without consensus on a single definition, in part due to its rapidly changing nature. Multiple definitions for AI are provided in the U.S. Code. For example, in 15 U.S.C. Section 9401(3), AI is defined as:

a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations or decisions influencing real or virtual environments. Artificial intelligence systems use machine and human-based inputs to-(A) perceive real and virtual environments; (B) abstract such perceptions into models through analysis in an automated manner; and (C) use model inference to formulate options for information or action.²

In a Senate Committee on Health, Education, Labor & Pensions (HELP Committee) white paper entitled Exploring Congress' Framework for the Future of AI: The Oversight and Legislative Role of Congress Over the Integration of Artificial Intelligence in Health, Education, and Labor, AI is defined as "computers, or computer-powered machines, exhibiting human-like intelligent capabilities."³ AI has also been broadly described as "computerized systems that work and react in ways commonly thought to require intelligence."⁴

Commonly referenced techniques to develop AI may include machine learning (ML), deep learning, supervised learning, and reinforcement learning, among others.⁵

The AI development life cycle in a health care setting may involve a variety of stakeholders, including AI developers, researchers, health systems, payers, patients, and federal agencies, among others.⁶ A number of federal agencies within the U.S. Department of Health and Human Services (HHS) have been involved in the regulation and use of AI, including the Assistant Secretary for Technology Policy/Office of the National Coordinator for Health Information Technology (ASTP/ONC)⁷ and the Centers for Disease Control and Prevention (CDC).⁸ CDC, for example, is exploring how AI technologies may be used to better estimate public health sentinel events, including suicide deaths.⁹ In December 2023, the Biden Administration announced the voluntary commitment of 28 health care companies, including providers and payers, to the responsible purchasing and use of AI in the health care setting.¹⁰ Companies that committed included Boston Children's Hospital, CVS Health, and Mass General Brigham, among others.¹¹ Additionally, several emerging private sector organizations are seeking to develop oversight measures and harmonized standards for AI use in health care, often in tandem with the public sector.¹²

This report provides a broad overview of the progress of AI technologies and selected landmarks in health care; discusses different AI techniques that are most often used in health care settings; explores primary health care tasks to which AI technologies are frequently applied; and provides an overview of selected federal efforts to oversee AI development, deployment, and monitoring in health care settings. Finally, this report provides Congress with legislative considerations regarding the use of AI technologies in health care. This report does not provide definitions for specific AI technologies and techniques.¹³

AI Evolution Broadly

While there has recently been an increased public focus on AI—particularly in response to the advances in, and widespread availability of, generative AI tools—AI has existed in some form since at least the 1950s.¹⁴ The progression of AI technologies may roughly be divided into multiple eras, though the exact chronology of AI development is the subject of much debate amongst experts and stakeholders.

One of the earliest recognized AI programs, written by Christopher Strachey in 1951, was designed to play the game checkers.¹⁵ In 1956, John McCarthy and several others introduced the term "artificial intelligence" at a Dartmouth College conference, marking what many consider to be the beginning of modern AI research.¹⁶ From the onset, health care was considered a promising area for AI application, and multiple clinical decision support (CDS) tools have been explored and developed since.¹⁷ For selected examples of these tools and others in health care, see Figure 1.

During the 1960s and 1970s, rule-based expert systems were the primary AI technologies focused upon. These models were limited by the computing power and data available at the time.¹⁸ A shift in focus to the development of ML techniques and neural network architectures in particular occurred during the 1980s and 1990s, producing machines able to "learn." An example of an AI technology from this period is IBM's Deep Blue, known in part for its defeat of world chess champion Garry Kasparov. In the 2000s advances in AI techniques, such as natural language processing, enabled the creation of personal assistants like Amazon's Alexa.¹⁹ In the last decade, deep learning techniques have also greatly improved when applied to tasks like image classification.²⁰

Early AI technologies like rule-based expert systems relied upon medical information supplied by, and rules curated by, medical experts. More recent ML algorithms have allowed for more sophisticated, ML-based models capable of recognizing patterns in vast data sets. The scale of such amounts of data may obscure the patterns from human analysts.²¹ The modern rapid advances in AI, and increased public scrutiny of such technologies, can be attributed to several intersecting factors, including increased availability of large-scale annotated clinical data sets, progress in ML techniques, the availability of open source ML packages, and increases in computer power and data storage, among others.²²

Figure 1. Selected AI Developments Timeline

Source: Figure adapted by CRS from various sources. See Shuroug A. Alowais et al., "Revolutionizing Healthcare: The Role of Artificial Intelligence in Clinical Practice," BMC Medical Education, vol. 23, no. 689 (2023), p. 2; Kun-Hsing Yu, Andrew L. Beam, and Isaac S. Kohane, "Artificial Intelligence in Healthcare," Nature Biomedical Engineering, vol. 2 (October 2018), pp. 719-720; Thomas Davenport and Ravi Kalakota, "The Potential for Artificial Intelligence in Healthcare," Future Healthcare Journal, vol. 6, no. 2 (June 2019), pp. 94-95, https://pmc.ncbi.nlm.nih.gov/articles/PMC6616181/pdf/futurehealth-6-2-94.pdf; Vivek Kaul, Sarah Enslin, and Seth A. Gross, "History of Artificial Intelligence in Medicine," Gastrointestinal Endoscopy, vol. 92, no. 4 (2020), pp. 807-809; Rahim Hirani et al., "Artificial Intelligence and Healthcare: A Journey Through History, Present Innovations, and Future Possibilities," Life, vol. 14, no. 5 (2024), pp. 2-3; OpenAI, "Introducing ChatGPT," November 30, 2022, https://openai.com/index/chatgpt/.

Overview of AI in Health Care

A variety of AI technologies may be used in health care settings. These AI technologies can be combined to perform more specialized tasks to support health care actors, often in the provision of care to patients.²³ The tasks to which AI technologies are applied in health care fall broadly into three categories: diagnosis and treatment, patient engagement and adherence with treatments plans, and administrative functions.

Techniques and Applications of AI Relevant to Health Care

A variety of AI techniques and applications may be used in health care settings. As is true in other sectors, ML techniques, including deep learning and neural networks, underpin many AI tools used in health care. Some of the more widely used AI techniques and applications in health care include natural language processing, which refers to the use of rule-based or ML approaches to understanding the structure and meaning of human language; rule-based expert systems; physical robots; and robotic process automation.

ML refers to a subset of AI techniques that rely on algorithms and large datasets to train computing devices to sort, identify, and predict information. One of the goals of ML is to create a model capable of interpreting previously unencountered data based upon the data it was trained on.²⁴ In health care, ML has wide applicability, and may, for example, be used in precision medicine to predict which treatment regimes will be most successful for individual patients.²⁵

Neural networks are a complex form of ML meant to imitate the way layers of neurons in the human brain process information.²⁶ In health care, neural networks are commonly used to interpret medical imaging, and may, for example, be used to detect the presence of diabetic retinopathy in images of the human eye.²⁷

Deep learning involves neural networks with many layers of artificial neurons, or nodes, leveraged to better predict outcomes.²⁸ In the health care context, deep learning may be used, for example, to diagnose cancerous lesions in radiological imaging.²⁹

Natural language processing attempts to parse human language and may involve speech recognition, text analysis, and translation, among other applications.³⁵ In health care, natural language processing may be used, for example, to organize and analyze documentation like clinical notes.³⁶

Rule-based expert systems essentially implement a series of "if-then" rules leading to a decision outcome. To create a rule-based expert system, a human expert is necessary to construct the rules and their relationships with one another.³⁷ In health care, rule-based expert systems are commonly integrated into EHRs and used to support health care providers in making clinical decisions like a patient's diagnosis.³⁸

AI can also be integrated into physical robots, which perform set tasks like lifting objects or delivering supplies. In health care, surgical robots are sometimes used to enhance physician capabilities. Conversely, robotic process automation refers to the performance of set digital tasks by computer programs, often for administrative purposes. In health care, robotic process automation may be used for tasks like billing.³⁹

While these AI technologies may be discrete, they are also increasingly used in combination with one another. For example, robotic process automation may leverage image recognition technologies to extract data from faxes to enter into medical transaction systems.⁴⁰

AI Applications in Health Care

As stated earlier, the use of AI in health care broadly falls into three categories: diagnosis and treatment, patient engagement and adherence with treatment plans, and administrative functions.

For patient diagnosis and treatment purposes, early AI technologies used were primarily rule-based expert systems. Since the 1970s, there was a shift from rule-based expert systems to more complex AI tools. AI technologies are often used to analyze medical imaging, like radiological or retinal images.⁴¹ Prediction models that detect and warn of high-risk conditions, like sepsis and heart failure, have also been developed.

AI technologies used for treatment and diagnosis purposes may also be used to support precision medicine.⁴² Precision medicine, also known as personalized treatment, is a developing medical approach tailored to specific patients using factors like genetics, environment, and lifestyle. Possible advantages of precision medicine include improved patient outcomes due to increased treatment effectiveness, efficiency, and safety. AI may potentially be useful in precision medicine because it can analyze large and complex datasets, predict outcomes, and optimize treatment plans. ML in particular may strengthen this process. For such treatments to be effective, these technologies need access to individuals' genotypic data, in part to ensure treatment decisions like medication selection and dosages are appropriate for each individual patient.⁴³

AI technologies can also be used to monitor and encourage patient engagement and adherence with medical treatment plans. AI-enabled systems, including those capable of sending alerts and targeted content to patients, could promote improved patient adherence to treatment plans. Wearable devices or patient EHRs may be used to collect data for such applications. Information from these sources can be shared with a patient's physician, who may in turn provide tailored recommendations to the patient (e.g., precision medicine).⁴⁴

AI technologies are also increasingly used in administration to increase efficiency in claims processing, clinical documentation, revenue cycle management, and medical records management. Additionally, AI-based chatbots can be used in telehealth services.⁴⁵

Selected Federal Actions

Many federal agencies are developing internal protocols for AI use; some are also developing policies shaping public AI use within their areas of authority.

Executive Order (E.O.) 14110

On October 30, 2023, the Biden Administration released E.O. 14110, entitled "Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence."⁴⁶ This E.O. includes direction for various federal agencies to develop, facilitate the development of, or promote standards for AI safety and security.⁴⁷ Further, according to the White House, the E.O. "protects Americans' privacy, advances equity and civil rights, stands up for consumers and workers, promotes innovation and competition, [and] advances American leadership around the world."⁴⁸ It specifically tasks the HHS Secretary (Secretary) with multiple initiatives, including the establishment of an HHS AI Task Force, the development of a quality strategy, the advancement of nondiscrimination protections, the creation of an AI safety program, and the development of a strategy for regulating the use of AI and AI-enabled tools in drug development processes.⁴⁹

Agency Actions

Multiple HHS offices and operating divisions, including the Assistant Secretary for Technology Policy/Office of the National Coordinator for Health Information Technology (ASTP/ONC), the U.S. Food and Drug Administration (FDA), the Office for Civil Rights (OCR), and the Centers for Medicare & Medicaid Services (CMS), have pursued regulatory actions regarding AI. These agency efforts are in early stages and are somewhat fragmented, though there is increasingly greater focus on unifying regulatory approaches across HHS.⁵⁰

Assistant Secretary for Technology Policy/Office of the National Coordinator for Health Information Technology

ASTP/ONC, a staff division within the Office of the Secretary, has pursued numerous AI initiatives. ASTP/ONC is generally responsible for advancing the development and use of health information technology (HIT), and establishing data sharing expectations.⁵¹ Additionally, in July 2024, HHS underwent a reorganization, during which ASTP/ONC was tasked with "oversight over technology, data, and AI policy and strategy."⁵² ASTP/ONC carries out its mission in part through its ONC Health IT Certification Program (Certification Program), under which HIT developers may voluntarily pursue ASTP/ONC certification of their HIT modules, including EHRs.⁵³ An HIT module certified under the program indicates the module meets baseline standardized criteria established by ASTP/ONC.⁵⁴

One of the challenges ASTP/ONC seeks to address related to AI is how to effectively capitalize on the potential of AI, ML, and related algorithmic technologies, while avoiding risks related to the use of invalid, inappropriate, unfair, or unsafe predictions.⁵⁵ ASTP/ONC notes that algorithms have been used in health care for decades in the form of rule-based decision support interventions (DSIs).⁵⁶

In January 2024, ASTP/ONC published a final rule in the Federal Register entitled "Health Data, Technology, and Interoperability: Certification Program Updates, Algorithm Transparency, and Information Sharing" (HTI-1).⁵⁷ In part, HTI-1 focuses on increasing algorithmic transparency through a revised Certification Program criterion for HIT modules that contain DSIs.⁵⁸ Under HTI-1, DSIs encompass both evidence-based and predictive algorithms.⁵⁹ For the purposes of ASTP/ONC's final rule, evidence-based DSIs are "limited to only those DSIs that are actively presented to users in clinical workflow to enhance, inform, or influence decision-making related to the care a patient receives and that do not meet the definition for [p]redictive DSI."⁶⁰ In turn, a predictive DSI is defined as a "technology that supports decision-making based on algorithms or models that derive relationships from training data and then produce an output that results in prediction, classification, recommendation, evaluation, or analysis."⁶¹

HTI-1 generally stipulates that certified HIT modules containing evidence-based or predictive DSIs must disclose to selected user parties particular source attributes, or categories of technical performance and quality information.⁶² The required source attributes for disclosure differ for evidence-based versus predictive DSIs.⁶³ The required disclosure is broadly intended to furnish selected users with bibliographic data for specified DSIs and information relevant to health equity, among other things, so that users may make more informed decisions when choosing and deploying DSIs in clinical settings.⁶⁴ HTI-1 also requires that developers keep source attributes up-to-date⁶⁵ and that developers implement risk management practices for predictive DSIs in particular.⁶⁶ Implementation of the portions of this rule described above are to take effect at the start of 2025.⁶⁷

Part of the voluntary commitment agreed to by the 28 health care companies regarding the responsible purchasing and use of AI in health care discussed previously draws upon HTI-1.⁶⁸ Specifically, the commitment stipulates that participants should "work with … peers and partners to ensure outcomes are aligned with fair, appropriate, valid, effective, and safe (FAVES) AI principles."⁶⁹ These FAVES AI principles are referenced in HTI-1 and are defined in the commitment statement as follows:

• Fair: "Outcomes of model do not exhibit prejudice or favoritism toward an individual or group based on their inherent or acquired characteristics."
• Appropriate: "Model and process outputs are well matched to produce results appropriate for specific contexts and populations to which they are applied."
• Valid: "Model and process outputs have been shown to estimate targeted values accurately and as expected in both internal and external data."
• Effective: "Outcomes of model have demonstrated benefit in real-world conditions."
• Safe: "Outcomes of model are free from any known unacceptable risks and for which the probable benefits outweigh any probable risk."⁷⁰

ASTP/ONC states that the finalized DSI certification criterion and accompanying transparency requirements in HTI-1 align with E.O. 14110 and will "improve transparency, promote trustworthiness, and incentivize the development and wider use of fair, appropriate, valid, effective, and safe [p]redictive DSIs to aid decision-making in healthcare."⁷¹ ASTP/ONC further asserts that the "resulting information transparency increases public trust and confidence in these technologies so that the benefits of [them] may expand in safer, more appropriate, and more equitable ways."⁷²

U.S. Food and Drug Administration

FDA regulates the safety and effectiveness of medical devices, which encompass "device software functions," including those that are AI/ML-enabled. The agency only has authority over certain software functions; specifically, those that meet the definition of "device" in the Federal Food, Drug, and Cosmetic Act (FFDCA).⁷³ The 21^st Century Cures Act (Cures Act, P.L. 114-255), enacted in 2016, further clarified the scope of software functions that meet the definition of device. Specifically, the Cures Act excluded from the definition of device those software functions intended: (1) "for administrative support of a health care facility;" (2) "for maintaining or encouraging a healthy lifestyle … unrelated to the diagnosis, cure, mitigation, prevention, or treatment of a disease;" (3) "to serve as electronic patient records;" and (4) "for transferring, storing, converting formats, or displaying" certain data and results.⁷⁴ The law also excluded CDS software that is for the purpose of displaying certain medical information about a patient (e.g., clinical practice guidelines); providing recommendations to a health care professional about prevention, diagnosis, or treatment of a disease; and enabling that professional to independently review the basis for the recommendations.⁷⁵ Determining whether a software function meets the definition of a device, and is thus regulated by FDA, can be a challenge for the developer. FDA has published several guidance documents addressing enforcement discretion policies for various types of device software functions (e.g., general wellness applications), generally based on the perceived risk of the product.⁷⁶

Device software functions regulated by FDA have proliferated in recent years, and include both software as a medical device (SaMD) and software integral to a traditional hardware device (or software in a medical device, SiMD).⁷⁷ AI/ML-enabled devices are more often SaMD, and FDA regulates SaMD as it does other devices. That is, these devices are subject to the same regulatory controls as other devices in the same risk class. This type of device is most often Class II—moderate risk—and therefore is subject to premarket review requirements, generally through a 510(k) notification, but also through a De Novo request if there is no predicate device available. In certain cases, AI/ML-enabled devices may be Class III, and would be subject to premarket approval (PMA) prior to marketing.⁷⁸ All devices are subject to general controls, including adverse event reporting, establishment registration and device listing, and the Quality System regulation (current good manufacturing practices), among others.⁷⁹

FDA maintains a list of AI/ML-enabled devices, based on publicly available information, that have received marketing authorization. As of August 2024, the agency reports that approximately 950 such devices have been authorized, cleared, or approved for marketing, with the majority in the field of radiology.⁸⁰ Most of these devices were cleared for marketing through the 510(k) premarket notification pathway, which requires a determination that the device is "substantially equivalent" to a legally marketed predicate device.

FDA has increasingly engaged on the issue of regulation of SaMD in recent years, and specifically AI/ML-enabled SaMD, publishing a proposed regulatory framework for AI/ML-enabled SaMD in 2019 and an action plan for AI/ML-enabled SaMD in 2021.⁸¹ In addition, the agency undertook a pilot program, the Software Precertification Pilot Program, which ran from 2017 until 2022.⁸² A key regulatory challenge in this area is oversight of a product that is not static, but rather changing over time, and how to predict, control, and mitigate that change over time, while preserving the safety and effectiveness of the device. To help address this issue, FDA introduced the concept of a "predetermined change control plan" (PCCP) in its 2019 proposed regulatory framework.⁸³ In late 2022, Congress passed the Food and Drug Omnibus Reform Act of 2022 (FDORA, P.L. 117-328, Division FF, Title III), which provided the agency with new statutory authority to approve a PCCP as part of a PMA application or a 510(k) notification.⁸⁴ FDA published draft guidance with more information about this mechanism and its appropriate inclusion in premarket applications in April 2023.⁸⁵

As issues with AI and ML span medical products and centers across FDA, in March 2024, the agency published "Artificial Intelligence and Medical Products: How CBER, CDER, CDRH, and OCP are Working Together."⁸⁶ This publication "outlines the agency's commitment and cross-center collaboration to protect public health while fostering responsible and ethical medical product innovation through Artificial Intelligence."⁸⁷

Office for Civil Rights

OCR administers various antidiscrimination statutes that apply to health care settings through actions including promulgating regulations and conducting administrative enforcement.⁸⁸ One such statute is Section 1557 of the Patient Protection and Affordable Care Act (ACA; P.L. 111-148), which prohibits discrimination on the basis of race, color, national origin, sex, disability, and age by certain health-related entities.⁸⁹ In May 2024, OCR issued new regulations implementing Section 1557.⁹⁰ Those regulations expressly clarify that Section 1557 bars covered entities from discriminating through the use of AI in patient care decisions.⁹¹ The rule reflects E.O. 14110's objective that AI not "deepen[] discrimination and bias" and its concern that "Artificial Intelligence systems deployed irresponsibly have reproduced and intensified existing inequities."⁹²

The Section 1557 rule may not apply to every use of AI in the patient-care context or to every entity that develops a health care-related AI. Section 1557 and its implementing rules apply to federally funded "health program[s] or activit[ies]," as well as to programs or activities "administered by an Executive Agency or any entity established under this title [Title I of the ACA]."⁹³ The statute does not define the phrase "health program or activity." HHS and courts generally agree that the statute covers many federal health programs HHS administers, as well as health plans sold through federal and state exchanges under the ACA's Title I.⁹⁴ There is also general agreement that the phrase includes providers such as hospitals, nursing homes, and physician practices.⁹⁵ The statute and its implementing rules may not apply to many software developers, and they do not apply to entities that do not receive federal funding.

The Section 1557 rule prohibits discrimination "through the use of patient care decision support tools,"⁹⁶ which are defined as "any automated or non-automated tool, mechanism, method, technology, or combination thereof used by a covered entity to support clinical decision-making in its health programs or activities."⁹⁷ OCR crafted the rule to account for providers' "widespread use of automated decision systems and AI, and the scale by which AI can influence covered entities' clinical decision-making."⁹⁸ The rule does not apply to AI use outside of "clinical decision-making affecting patient care."⁹⁹ For example, it does not apply to tools used for scheduling or billing.¹⁰⁰ OCR requested additional comment on whether "decision support tools" used in nonclinical settings warrant further rulemaking.¹⁰¹ OCR's definition of "patient care decision support tools" overlaps somewhat with the definitions for predictive DSIs and evidence-based DSIs in ASTP/ONC's HTI-1 final rule, although the Section 1557 regulation applies to health care providers using AI tools (among other things), while ASTP/ONC's regulation focuses on developers.¹⁰²

To combat discrimination through the use of patient care decision support tools, the Section 1557 rule specifies that "a covered entity has an ongoing duty to make reasonable efforts to identify" any such tools it uses that "measure race, color, national origin, sex, age, or disability."¹⁰³ A covered entity must "make reasonable efforts to mitigate the risk of discrimination" from tools using such inputs.¹⁰⁴ In its responses to public comments on the proposed rule, OCR "acknowledge[d] that it is not always possible to completely eliminate the risk of discriminatory bias in patient care decision support tools."¹⁰⁵ One commenter observed in evaluating the proposed rule that "[b]ecause the inner workings of an algorithm can be difficult to fully understand," it "puts the burden on clinicians to be well-versed enough in data and computer science to evaluate, oversee, and correct for potential biases in algorithms."¹⁰⁶

Centers for Medicare & Medicaid Services

CMS administers the Medicare program, including Medicare Advantage (MA, or Medicare Part C). MA is an alternative to original Medicare under which eligible beneficiaries may receive their covered services through health care companies under contract with CMS. As referenced in the HHS factsheet, "Biden-Harris Administration Announces Voluntary Commitments from Leading Healthcare Companies to Harness the Potential and Manage the Risks Posed by AI,"¹⁰⁷ CMS recently issued a final rule regarding how MA plans may use prior authorizations, which also discusses the limits of using AI in prior authorizations and other coverage decisions.¹⁰⁸ This subsection briefly describes Medicare, MA, coverage requirements, payments to MA plans, prior authorizations, and the recent rule.

Background on Medicare, MA, and Coverage Requirements

Medicare is a federal program that pays for covered health care services¹⁰⁹ of qualified beneficiaries.¹¹⁰ It consists of four parts – A through D. Parts A and B cover a host of services including hospital care, skilled nursing care, and physician services. Part D covers outpatient prescription drugs.¹¹¹ Part C¹¹² is an alternative to Parts A and B, that covers substantially all required Parts A and B services.¹¹³ MA plans may also include an integrated Part D benefit.

Medicare Parts A and B are referred to as "original Medicare" or "fee for service (FFS)" Medicare because they pay providers for each item, service, or spell of illness that is treated.

Medicare covers a broad range of medical items and services, but there are limits. In general, Medicare pays for items or services that (a) fit into a Medicare benefit category (such as inpatient hospital care, or physician services),¹¹⁴ (b) are not statutorily excluded,¹¹⁵ and (c) are reasonable and necessary.¹¹⁶ The HHS Secretary, and contractors referred to as Medicare Administrative Contractors (MACs), have discretion to determine what specific items and services are "reasonable and necessary" and under what clinical conditions they will be covered. These policies are referred to as National Coverage Determinations (NCDs) when created by the HHS Secretary, or Local Coverage Determinations (LCDs) when created by the MACs. NCDs and LCDs are developed through processes that include examination of peer-reviewed medical research, consensus statements from medical societies, and input from the public.¹¹⁷ In addition, MACs can make coverage determinations on a case-by-case basis considering the clinical circumstances of individual beneficiaries.¹¹⁸

In general, the coverage determinations that apply to FFS Medicare also apply to MA plans.¹¹⁹ MA plans have long been subject to sub-regulatory guidance that requires them to make coverage determinations that are no more restrictive than the NCDs and LCDs under original Medicare.¹²⁰ However, recent HHS Office of Inspector General (OIG) research found that MA enrollees were being denied coverage of items and services that would have been covered had the beneficiary been enrolled in original Medicare.¹²¹ Based partly on these findings, CMS engaged in rulemaking to clarify the requirement that items and services covered under original Medicare must also be covered by MA plans.¹²² The next section describes MA payments and the incentives to deliver care efficiently.

MA Payments and Prior Authorizations for Care

Rather than being given the fee for service payments used in original Medicare, MA plans receive a capitated payment regardless of the number of covered items or services an enrollee uses.¹²³ As such, MA plans are at risk if total expenditures exceed payments; and within limits, MA plans may retain payments that exceed health care expenditures.¹²⁴ Partly due to the capitated payments, MA plans have a financial incentive to provide covered services efficiently.

To achieve this, MA plans employ various utilization management (UM) techniques, such as prior authorization, to ensure that items and services an enrollee uses and the plan covers are reasonable and necessary. Prior authorizations¹²⁵ can reduce unnecessary or potentially harmful care, improper payments, and overall expenditures; they may also be burdensome and costly to providers, and delay or discourage medically necessary care.¹²⁶ It is unclear how many MA plans use AI as part of the prior authorization process (or as part of another UM technique) to, for example, expedite approval or disapproval of prior authorizations, but the use of AI in MA prior authorizations was discussed in recent rulemaking.¹²⁷

MA Rulemaking Including Comments on AI

In a 2023 final rule, CMS established guardrails "to ensure that utilization management tools are used, and associated coverage decisions are made, in ways that ensure timely and appropriate access to medically necessary care for beneficiaries enrolled in MA plans."¹²⁸

The rule codified the sub-regulatory requirement mentioned above that MA plans must base coverage determinations on the Medicare statute, regulations, NCDs, and LCDs.¹²⁹ It also outlined circumstances and processes under which MA plans may establish additional coverage criteria when coverage criteria from the required sources are not fully established.¹³⁰ Starting in contract year 2024, MA plans must establish Utilization Management Committees with specified composition.¹³¹ These committees are required to review all UM policies at least annually.¹³² Prior authorizations may only be used to confirm the presence of a diagnosis or other medical criteria that are the basis for the coverage determination.¹³³ MA organizations must make medical necessity determinations based on, among other things, the enrollee's medical history, physician recommendations, and medical notes.¹³⁴ And if an MA plan intends to issue a partially or fully adverse determination, the determination of coverage must be reviewed by a physician or other medical professional with expertise in the relevant field of medicine.¹³⁵

The rule does not refer to "AI" or "Artificial Intelligence" directly, but in response to some comments during the rulemaking process, CMS mentioned two digital tools—InterQual and MCG systems—both of which use AI.¹³⁶ Some commenters requested that CMS prohibit the use of certain tools, such as InterQual or MCG systems, while others advocated for continued use of such tools to guide medical necessity determinations.¹³⁷ CMS clarified that using these tools in isolation is prohibited, as it would violate other regulatory requirements, such as the requirement that if an organization expects to issue a partially or fully adverse determination, the decision must be reviewed by a physician or other appropriate health care professional with expertise in the relevant field of medicine or health care.¹³⁸ CMS acknowledged that these products, along with others, are designed to assist in the coverage determination process by consolidating clinical data, medical literature, and CMS policies. However, use of the tools does not absolve an MA plan from adhering to other regulatory requirements. For example, if the MA plan establishes additional internal coverage criteria, those criteria must be based on publicly available evidence in widely used clinical guidelines or medical literature, and the plan must disclose the factors considered in formulating the coverage policy.¹³⁹ Furthermore, as noted above, if an MA plan intends to issue a partially or fully adverse determination of coverage, the determination must be reviewed by a physician or other medical professional with relevant expertise.¹⁴⁰ In short, MA plans may use algorithms, software, or AI systems to assist in coverage determinations,¹⁴¹ "[h]owever, use of these tools, in isolation, without compliance with requirements in this final rule…is prohibited."¹⁴²

As AI becomes more integrated into different aspects of health care, policies that do not mention AI directly may nonetheless affect its use.

Selected Issues for Consideration

As AI technology use expands in the health care setting, certain existing challenges may be exacerbated, while novel ones may emerge.¹⁴³

As a foundational issue, trust is required for the effective application of AI technologies. In the clinical health care context, this may involve how patients perceive AI technologies. In a 2023 survey of the U.S. public conducted by the Pew Research Center, 60% of respondents replied it would make them uncomfortable if their care provider relied on AI when making decisions like diagnoses and treatment recommendations. People's perceptions of AI use in health care may be mediated by factors like their genders, races/ethnicities, ages, education levels, and familiarity with AI.¹⁴⁴

Additional challenges with AI use in health care may be associated with issues of (1) data access, (2) bias, (3) lack of transparency, (4) privacy, (5) scaling and integration, and (6) uncertainty over liability.¹⁴⁵

Though health data have proliferated in recent years, it may be difficult for developers to access the large volumes of high-quality data needed to create effective health care AI tools.¹⁴⁶ High-quality data may be difficult to consolidate, since medical data can be fragmented and stored and formatted differently. Additionally, there may be disagreement over who owns, or should own, the medical data used to create these AI tools.¹⁴⁷

In turn, the data used to develop AI tools may be limited or biased; this can reduce the safety of such AI tools, and may make them less effective for different patient populations, resulting in treatment disparities in terms of access to health care services and quality of care and health outcomes.¹⁴⁸ Bias may be introduced into data in a variety of ways, including poorly developed study designs to collect data (e.g., nonrepresentative data samples) and nonstandardized data across sources.¹⁴⁹

AI tools may also lack transparency. This is in part because of the inherent difficulty in determining how some AI tools work (i.e., the black box problem). However, a lack of transparency may also stem from more controllable factors, like the lack of AI tool evaluations in clinical settings or developers not making their tools open-source. This lack of information can make it difficult for health care providers to evaluate whether an AI tool is appropriate for a specific health care application.¹⁵⁰ It may also make it difficult for providers to follow rules regulating the use of AI, such as OCR's Section 1557 rule.

There may be challenges associated with privacy and AI use. As AI systems are developed, increasingly large quantities of data will likely be accessible to more people and organizations.¹⁵¹ This can add to privacy risks and concerns regarding health data. Privacy concerns often dovetail with security challenges as well. This is because, as digital health technologies like AI become more complex, greater cybersecurity risks are introduced.¹⁵²

AI tools can also be challenging to scale up and integrate into new settings because of differences among institutions and patient populations. For example, what is effective in a small rural clinic may not be as effective in a larger metropolitan hospital. Additionally, costs associated with scaling and integrating AI technologies can be prohibitive for some health care institutions.¹⁵³

The number of parties involved in developing, deploying, and using AI tools has made it difficult to determine legal liability associated with these technologies. Limited legal precedent further compounds these challenges. Such uncertainty may foster reluctance to adopt such technologies, and could slow overall innovation.¹⁵⁴

AI use in health care may also prompt more general considerations regarding AI regulatory harmonization nationally and internationally, as well as issues related to the environmental impact of AI use.

A number of frameworks for responsible AI use and regulation have been proposed by both public and private entities.¹⁵⁵ While these proposals may confront the key issues regarding the responsible use of AI, there appears to be an increasing need for a consensus framework that stakeholders may reference to facilitate both cohesive regulation and interoperability, especially in health care.¹⁵⁶

Internationally, there has been a similar proliferation of AI guidance and regulatory approaches. AI technologies created and used in America will likely be used both domestically and abroad, and AI technologies developed and used internationally will likely be used in the United States. It is seen as increasingly necessary to consider and be aware of how an AI technology developed in one country may be regulated in another.¹⁶¹

Finally, related to the use of AI generally is its potential impact on the environment. The high demands on certain resources (e.g., minerals, water, and electricity) to build and power AI technologies could present a risk to human health.¹⁶² AI computing and storage functions especially create large energy demands on data centers, which in turn can lead to increased global emissions.¹⁶³ Thus, the efficiency and environmentalism of AI technologies and infrastructure may be of particular importance.¹⁶⁴

Conclusion

AI technologies have existed for decades, and these technologies have become more accessible to the public recently due to technological advancements like generative AI. As their applications have expanded in recent years, the use of AI technologies in health care settings has been subject to greater scrutiny by stakeholders. Though AI technologies may prompt advances and improvements in health care services, many stakeholders maintain there is a need for regulatory guardrails to address potential challenges associated with trust, data access, bias, lack of transparency, and privacy. Government agencies and other stakeholders have begun the process of developing such guardrails. The potential need for harmonization amongst party efforts and who should lead such harmonization initiatives (e.g., public versus private groups, or perhaps public-private partnerships) are among the emerging considerations.

Footnotes