ChatGPT Health's trust patterns for sensitive AI

Trust is a control problem, not a model property

A health answer can be accurate and still feel unsafe. In a 2024 study, Benda and colleagues found that 57.8% of participants were uneasy with AI that was highly accurate but not explainable, and 76.6% were uneasy with AI that was 90% accurate but not explainable. OpenAI announced that ChatGPT Health would open to U.S.-based users over 18 on all plans. The launch matters because it puts health data behind visible controls.

The discomfort is not new. Pew Research Center data from 11,004 U.S. adults showed that 60% felt uneasy about clinicians using AI to make diagnosis or treatment decisions. Riedl and colleagues showed that trust, information sharing, treatment adherence, and satisfaction declined as the care model moved from human doctors to AI-assisted doctors to AI operating alone.

Users do not need a more confident model. They need a legible set of controls: what the product can read, what it may use, what it will not do, and how the user can undo it.

Six visible controls make sensitive AI legible

ChatGPT Health supports connections to health providers and to consumer health services including Apple Health, Weight Watchers, MyFitnessPal, and Function. It also supports integration of medical records from hospital systems such as Epic and Oracle Health, and from health platforms such as One Medical and Function Health. Users can review and change which categories of health data are connected to ChatGPT, including post-visit notes, care-team notes, lab results, and medications.

ChatGPT requests user permission before using connected health information to personalize a response, and Health-connected data gets extra encryption safeguards. OpenAI stated that records shared with ChatGPT are not used for model training or ad targeting. OpenAI cautioned that ChatGPT is not a physician and should not replace clinical care, diagnosis, or treatment.

• Ask before the model uses connected data. A per-request permission prompt should appear when the answer will draw on private records.
• Let users choose data categories, not just apps. Controls should separate the data categories the user connected.
• Make every source revocable. A connected health app or provider record should be removable without support tickets.
• State the professional limit in plain language. The product should say it is not a clinician and not a substitute for care.
• Promise no training and no ad targeting. The data-use statement should say shared records will not train models or target ads.
• Disclose extra encryption for sensitive connections. Security notes should say sensitive connections receive extra encryption safeguards.

A health answer that silently uses a private record feels like a breach even when the answer is helpful. Users think in kinds of information, not app names, so category controls should separate the data they connected. A connection made in a moment of curiosity can outlive the intent behind it, so revocation should be immediate.

Plain-language limits keep the product from sounding like a clinician and inviting the user to outsource judgment, while a no-training promise stops a record shared for one answer from becoming input for the next model. Encryption notes let users verify that the product treats the data as sensitive.

Ship the limits before the model gets more capable

Those controls are part of the interface. When a product uses sensitive personal data, the model’s answer is only one part of the experience. The other part is the user’s ability to inspect the permission, revoke the source, and understand the limit. If the answer is wrong, a visible control makes the failure easier to explain. If the answer is right, the same control makes the product feel safe enough to use again.

A useful test is to ask a new user to find the off switch. If the control is in a settings menu, that is good. If it is in a help article, that is a gap. Put the control where the user will look first, and keep the limit visible in the same place.