The U.S. Department of Health and Human Services is moving fast to put artificial intelligence agents inside the medical system. But the speed of the rollout is raising concerns that safety and evidence are being left behind.
## The Federal Push: From Chatbots to Prescribing
Multiple federal initiatives are underway. The FDA has already approved more than 1,500 medical devices that include AI, performing discrete tasks like identifying tumors or blood clots . But those tools are not the same as the newer generation of chatbots and **agentic AI** systems—models that can perform tasks traditionally handled by physicians, including prescribing medications .
The FDA does not yet have rules designed for these systems . The agency is working to revise its regulations to keep pace with more autonomous AI .
Meanwhile, Medicare is exploring how to pay for AI tools that provide care or diagnosis. Officials have discussed whether **AI doctors** operated by technology companies should be paid as much as **60% to 80%** of what human physicians earn for the same service . Medicare has allowed more than **200 companies** to participate in pilots that can include AI .
One federal initiative involves an AI chatbot providing therapy to Medicare beneficiaries with depression or anxiety . Another, funded by the Advanced Research Projects Agency for Health (ARPA-H) with about **$60 million**, aims to deploy autonomous AI to manage care for heart failure patients, including a chatbot that could prescribe and manage medications . Under current rules, a physician would oversee the AI .
## The Venture Capital Connection
The rollout has raised questions about the influence of Silicon Valley investors who have traditionally played little role in federal health policy .
Billionaire venture capitalist **Vinod Khosla** has been particularly visible in conversations with senior officials, including CMS Administrator Mehmet Oz . Khosla has aggressively promoted his son's AI startup, **Curai Health**, which offers AI primary care services . He has declared that "**the era of human doctors is over, AI will simply do it better**" .
The FDA has also signaled openness to working more directly with venture capital firms. In December 2025, the agency issued a Request for Information inviting VC firms to propose a new contracting model that would allow their portfolio companies to compete for federal task orders .
## The Medical Establishment Pushes Back
The American Medical Association has warned that models designed to independently provide patient care are **not ready for widespread use** . AMA CEO John Whyte has questioned the evidence cited by AI proponents, noting that much of it is based on **simulations** rather than real-world outcomes .
"The AI in healthcare should not be driven by the tech industry, but led by the clinical medical community," Whyte said. "Right now, it's technology that's leading everything" .
The AMA has urged Congress to establish clear boundaries prohibiting AI chatbots from diagnosing or treating mental health conditions, and to require transparency about whether users are interacting with a machine . It also wants limits on advertising within mental health chatbots and stronger privacy protections .
A health department spokesperson said the department is taking a "**responsible, evidence-based approach**" to AI and expanding its use as evidence demonstrates value .
## Utah's Autonomous Prescribing Experiment
The most advanced experiment is happening at the state level. In January 2026, Utah launched a first-of-its-kind pilot program allowing an autonomous AI agent developed by **Doctronic** to renew prescriptions for patients with chronic conditions .
Under the agreement, physicians review the AI's output for the first **250 patients** before the system begins acting autonomously. The next **1,000 patients** are reviewed retrospectively . The program covers **192 drugs** for conditions like hypertension, depression, and diabetes .
But legal scholars have raised serious questions. A paper published in the *New England Journal of Medicine* argued that the tool may not have undergone FDA premarket review, and that federal law requires a licensed human "prescriber" to authorize refills .
"The plain text interpretation of that is it needs to be a licensed human physician, not an AI tool," said Sara Gerke of the University of Illinois College of Law . "What's happening in Utah might be a violation of the law that the FDA has so far taken no enforcement action on" .
## What This Means for Patients
For American patients, the rapid deployment of medical AI raises practical concerns.
**Evidence gap**: Much of the evidence supporting AI's superiority over physicians comes from simulations, not real-world clinical trials .
**Liability questions**: When an AI chatbot fails to escalate a dangerous situation, who is responsible—the developer, the hospital, or the physician who signed off? The AMA warns that current frameworks lack clear accountability .
**Privacy risks**: Mental health chatbots collect highly sensitive data. The AMA has called for limits on data collection, meaningful deletion rights, and safeguards against unauthorized sharing .
The federal government is betting that AI can expand access to care, particularly in rural areas with physician shortages . Whether it can do so safely, with adequate oversight and evidence, remains an open question.
---
**Frequently Asked Questions (FAQs)**
**1. What federal agencies are deploying medical AI?**
The FDA, CMS, and ARPA-H are all involved. The FDA has approved over 1,500 AI-enabled medical devices. Medicare is exploring payment models for AI-provided care and running pilots with over 200 companies. ARPA-H is funding autonomous AI for heart failure management .
**2. What is "agentic AI" in healthcare?**
Agentic AI refers to systems that can take actions autonomously—such as prescribing medications or managing treatment—rather than just providing information or recommendations .
**3. What concerns has the AMA raised?**
The AMA says models designed to independently provide care are not ready for widespread use. It questions evidence based on simulations and urges Congress to prohibit AI chatbots from diagnosing or treating mental health conditions, require transparency, and establish privacy safeguards .
**4. What is the Utah pilot program?**
Utah allowed the startup Doctronic to deploy an autonomous AI agent that renews prescriptions for chronic conditions. Physicians review the first 250 patients, then the AI operates autonomously with retrospective review of the next 1,000 .
**5. Is autonomous AI prescribing legal?**
Legal scholars argue it may violate federal law, which requires a licensed human "prescriber" to authorize refills. The FDA has taken no enforcement action, but the issue remains unresolved .
**6. What is Vinod Khosla's role?**
Khosla is a billionaire venture capitalist who has been highly visible in conversations with senior federal health officials. He has promoted his son's AI healthcare startup and declared that the era of human doctors is over .
**7. How is Medicare planning to pay for AI care?**
Officials have discussed paying AI doctors **60% to 80%** of what human physicians earn for the same service. Medicare is also exploring a new payment category for AI software that supports diagnosis or care .
---
**Disclaimer**
*This article is for informational and educational purposes only and does not constitute medical, legal, or professional advice. The information provided is based on publicly available reports from the New York Times, Becker's Hospital Review, the Journal of the American Medical Association, and other cited sources as of September 2026. The deployment of AI in healthcare is rapidly evolving, and the safety and efficacy of these tools remain under investigation. Before making any medical decisions based on the content of this article, please consult with qualified healthcare professionals who can evaluate your specific situation.*
Posts
No comments:
Post a Comment