Warning: Don't Give Your Health Data to Chatbots

ChatGPT Health makes a bold promise of privacy, yet ultimately, users must rely solely on OpenAI's assurances.

According to OpenAI, over 230 million individuals seek health and wellness advice from ChatGPT weekly. The company suggests many view the chatbot as a valuable "ally" for navigating complex insurance procedures, completing paperwork, and enhancing self-advocacy in their healthcare journey. In return, OpenAI hopes users will entrust its chatbot with highly sensitive medical data, including diagnoses, medications, and test results. However, while engaging with a chatbot may increasingly resemble a doctor's consultation, it lacks the same legal framework. Unlike medical providers, technology companies are not bound by identical obligations. Experts advise careful consideration before sharing personal health records with these platforms.

The health and wellness sector is rapidly emerging as a critical arena for AI development, serving as a significant test of user willingness to integrate these systems into their daily lives. This month, two prominent industry players significantly advanced their medical offerings. OpenAI launched ChatGPT Health, a dedicated feature within ChatGPT, designed for health-related inquiries in what it claims is a more secure and personalized environment. Concurrently, Anthropic unveiled Claude for Healthcare, a "HIPAA-ready" solution marketed for use by hospitals, healthcare providers, and consumers alike. Notably absent from these direct consumer pushes is Google, despite its Gemini chatbot being a leading AI tool, though the company did announce an update to its MedGemma medical AI model for developers.

OpenAI actively encourages ChatGPT Health users to share sensitive details such as medical records, lab results, and wellness data from integrated apps like Apple Health, Peloton, Weight Watchers, and MyFitnessPal, promising deeper insights in return. The company explicitly guarantees that health data will remain confidential, will not be used for AI model training, and that robust measures are in place to ensure data security and privacy. Furthermore, OpenAI states that ChatGPT Health conversations will be isolated within a distinct section of the app, allowing users to review or delete their Health "memories" at any time.

OpenAI's pledges to safeguard sensitive user data have been inadvertently bolstered by the simultaneous launch of a similarly named product featuring stricter security protocols. This tool, ChatGPT for Healthcare, is part of a broader enterprise suite designed to support businesses, hospitals, and clinicians directly. Its suggested applications include streamlining administrative tasks like drafting clinical letters and discharge summaries, and assisting physicians in compiling the latest medical evidence for improved patient care. Unlike consumer-facing offerings, especially free tiers, these enterprise-grade products come with enhanced protections and are designed to comply with medical sector privacy obligations. Given the close proximity of their launch dates and similar nomenclature—ChatGPT for Healthcare was announced just a day after ChatGPT Health—it is easy to confuse the two and mistakenly assume the consumer product benefits from the same level of protection as its clinically oriented counterpart. Indeed, many individuals interviewed for this story made precisely this error.

Even if a company pledges to protect your data, it could always change its mind.

Regardless of any security assurances, their robustness is far from absolute. Users of tools like ChatGPT Health often have minimal safeguards against data breaches or unauthorized use beyond the stipulations in terms of service and privacy policies, experts inform The Verge. With most states lacking comprehensive privacy laws, and no overarching federal privacy legislation, data protection for AI tools like ChatGPT Health "largely depends on what companies promise in their privacy policies and terms of use," states Sara Gerke, a law professor at the University of Illinois Urbana-Champaign.

Even with trust in a company's commitment to data protection—OpenAI claims to encrypt Health data by default—the terms can be altered. Hannah van Kolfschooten, a researcher in digital health law at the University of Basel in Switzerland, explains, "While ChatGPT does state in their current terms of use that they will keep this data confidential and not use them to train their models, you are not protected by law, and it is allowed to change terms of use over time." She adds, "You will have to trust that ChatGPT does not do so." Carmel Shachar, an assistant clinical professor of law at Harvard Law School, echoes this sentiment: "There’s very limited protection. Some of it is their word, but they could always go back and change their privacy practices."

Assurances of compliance with healthcare data protection laws, such as the Health Insurance Portability and Accountability Act (HIPAA), should also offer limited comfort, according to Shachar. While HIPAA provides valuable guidance, she explains, there are minimal consequences if a company that voluntarily complies fails to uphold its standards. Voluntary compliance is distinctly different from being legally bound. "The value of HIPAA is that if you mess up, there’s enforcement."

Medicine is a heavily regulated field for good reason.

The concerns extend beyond just privacy. Medicine is a heavily regulated field precisely because errors can have dangerous, even fatal, consequences. Numerous instances highlight chatbots confidently disseminating false or misleading health information. For example, one individual developed a rare condition after ChatGPT suggested replacing salt in his diet with sodium bromide, a substance historically used as a sedative. Similarly, Google’s AI Overviews incorrectly advised pancreatic cancer patients to avoid high-fat foods, directly contradicting recommended dietary practices.

To mitigate these risks, OpenAI explicitly states that its consumer-facing tool is intended for use in close collaboration with physicians and is not designed for diagnosis or treatment. Tools specifically engineered for diagnosis and treatment are classified as medical devices, subjecting them to far stricter regulations, including mandatory clinical trials to prove efficacy and ongoing safety monitoring post-deployment. Despite OpenAI's clear awareness that supporting user health and well-being is a primary use case for ChatGPT—evidenced by 230 million weekly health-related queries—the company's assertion that its tool is not a medical device carries significant weight with regulators, as explained by Gerke. She notes, "The manufacturer’s stated intended use is a key factor in the medical device classification," meaning companies that disclaim medical use can largely circumvent oversight, even if their products are, in practice, being used for medical purposes. This highlights the profound regulatory challenges posed by emerging technologies like chatbots.

For the time being, this disclaimer keeps ChatGPT Health outside the regulatory scope of bodies like the Food and Drug Administration. However, van Kolfschooten posits that it is entirely reasonable to question whether such tools should indeed be classified and regulated as medical devices. She emphasizes the importance of examining actual usage patterns, not just company statements. OpenAI, in its product announcement, suggested ChatGPT Health could be used to interpret lab results, track health behaviors, or assist in reasoning through treatment decisions. If a product performs these functions, it could reasonably be argued to fall under the U.S. definition of a medical device, she suggests, hinting that Europe's more robust regulatory framework might be why the product is not yet available in that region.

"When a system feels personalized and has this aura of authority, medical disclaimers will not necessarily challenge people’s trust in the system."

Despite disclaiming ChatGPT's use for diagnosis or treatment, OpenAI has invested substantial effort in demonstrating its medical capabilities and encouraging users to leverage it for health inquiries. The company prominently featured health as a major application during the launch of GPT-5, with CEO Sam Altman even inviting a cancer patient and her husband on stage to discuss how the tool aided her in understanding her diagnosis. OpenAI claims to evaluate ChatGPT's medical proficiency against its self-developed benchmark, HealthBench, which involves over 260 physicians across numerous specialties and "tests how well AI models perform in realistic health scenarios," although critics note its lack of transparency. Other studies, often small, limited, or conducted internally by the company, also hint at ChatGPT's medical potential, indicating its capacity to pass medical licensing exams, improve patient communication, outperform doctors in diagnosis, and help physicians reduce errors when used as a supportive tool.

Van Kolfschooten argues that OpenAI’s active promotion of ChatGPT Health as an authoritative source of health information could effectively undermine any disclaimers advising users against its medical application. She states, "When a system feels personalized and has this aura of authority, medical disclaimers will not necessarily challenge people’s trust in the system."

Companies like OpenAI and Anthropic are banking on this trust as they compete for dominance in what they perceive as the next significant AI market. The substantial number of individuals already utilizing AI chatbots for health-related purposes suggests they may be tapping into a genuine need. Given prevalent health inequalities and the widespread challenges in accessing even basic care, this development could potentially be beneficial. However, its positive impact hinges on whether this trust is indeed well-placed. We entrust our private information to healthcare providers because the profession has demonstrably earned that trust. It remains uncertain whether an industry renowned for its "move fast and break things" philosophy has yet earned the same level of confidence in such a sensitive domain.