However, even when the researchers typed in medical scenarios directly, the chatbots had difficulty determining when symptoms required immediate medical attention or just nonurgent care. There were also several occasions when the chatbots fabricated information, such as the number of emergency hotlines, or changed its answers significantly based on just slight variations in a question.

For example, a participant saying they had a "terrible headache," along with a stiff neck, led to the chatbot treating the problem as a minor health issue. However, when a participant said they had "the worst headache ever" and a stiff neck, the chatbot recommended they go to the ED or call emergency services.

"Very, very small words make very big differences," said Andrew Bean, a graduate student at the University of Oxford and the study's lead author. 

According to Robert Wachter, chair of the department of medicine at the University of California, San Francisco, "[t]here's a lot of cognitive magic and experience that goes into figuring out what elements of the case are important that you feed into the bot." While doctors are trained to recognize which details are relevant and which can be ignored, the general public is not, which makes it more difficult for them to use AI chatbots for health advice effectively.

However, instead of relying on people to create the perfect question for chatbots, Bean said chatbots should be programmed to ask follow-up questions similar to the ways doctors gather information from patients when making a diagnosis.

"Is it really the user's responsibility to know which symptoms to highlight, or is it partly the model's responsibility to know what to ask?" Bean asked.

Currently, the researchers are planning to conduct similar studies in different countries and languages and want to see if these changes will impact AI's performance over time. "We hope this work will contribute to the development of safer and more useful AI systems," Bean said.

How AI chatbots will impact the future of healthcare

By Carol Chouinard and Ben Isenhour

The growing prevalence of AI chatbots in healthcare

Over the last few years, some patients have lost trust in the health system. Because of this, more people are turning to alternative sources for healthcare guidance, including AI chatbots like ChatGPT and their own online research.

As technology improves, patients are gaining access to more methods for gathering medical advice, which is changing the way they engage with the health system. According to several provider surveys, patients are now coming to their providers with more specific questions and expectations.

AI chatbots will continue to change how patients interact with their own healthcare, providing them with a more interactive and personalized experience. Although most AI chatbots generally rely on publicly available data, there are ongoing efforts to integrate patients' medical records and other personal information to better inform the chatbots' responses. For example, OpenAI recently released a new health-focused feature called ChatGPT Health, which allows users to upload their medical records and connect data from wellness apps.

At the same time, there have been growing policy efforts to give patients more access and control over their health data, both in the United States and abroad. Currently, several technology companies, including Epic, Google, OpenAI, Anthropic, and Microsoft, are expanding their healthcare capabilities with the help of AI.

All of these changes theoretically allow consumers to gain more control over their health data and use it to improve their health and other aspects of their lives. Going forward, providers will need to address this shift by finding new ways or channels to influence healthcare behaviors and support care management.

The shortcomings of AI chatbots in healthcare

Despite widespread interest in AI chatbots and other tools, it's important to not overlook their risks. Many AI tools are prone to faulty or misleading information, which can negatively impact patients' health.

Organizations will need to determine how to minimize the risks associated with patients using third-party AI tools and help patients use them responsibly. Organizations that deliver a clinically governed digital front door that's grounded in validated content, tuned to local care pathways, and designed to route people safely (self-care vs. primary care vs. urgent care/ED) to clinicians with clear escalation criteria will likely be the most successful.

The best use of AI in healthcare

While today's results highlight real limitations, they also reflect how early we are in the evolution of these tools. AI chatbots are improving rapidly and will likely look very different in the future. For organizations with the resources to do so, experimenting with multiple tools — and continuing to engage with them as they develop — may offer more value than depending on any one platform today.

From Optum's perspective, the greatest value of generative AI appears when it serves as a copilot. It should help members and patients ask better questions, understand their choices, and take appropriate next steps. It should also help clinicians reduce cognitive and administrative burden rather than serve as an autonomous clinical decision-maker.

*Advisory Board is a subsidiary of Optum. All Advisory Board research, expert perspectives, and recommendations remain independent.