AI is increasingly being used to reduce administrative burdens in healthcare, so that healthcare professionals have more time for direct patient care. However, a large-scale study by Dartmouth shows that AI does not always deliver the expected gains in efficiency and time when responding to messages from patients via online patient portals. In many cases, correcting AI-generated texts actually takes doctors more time than if they were to write the report or email themselves.
The researchers presented their findings at the 2026 Annual Meeting of the Association for Computational Linguistics. For the study, nearly 146,000 messages exchanged between 10,105 patients and their GPs within Dartmouth Health were analysed. The researchers developed a method to systematically compare AI-generated responses with those actually sent by doctors. According to the researchers, whilst AI can write like a doctor, the system does not yet think like a clinician. As a result, the suggested responses often fail to align adequately with the medical judgements doctors make in practice.
Long, incorrect responses
The researchers tested six commercial language models, including ChatGPT, Gemini, Claude, Llama, Aloe and Qwen. The analysis shows that AI-generated responses are often too long, fail to ask sufficiently relevant follow-up questions and regularly contain incorrect or superfluous medical information. An example from the study illustrates this difference. In the case of a 32-year-old woman who was taking antacids and complaining of persistent nausea, the AI suggested she adjust her diet because of the medication. The treating doctor replaced this with a simple, yet clinically far more relevant, question: whether she might be pregnant.
According to the researchers, it is precisely such minor adjustments that can result in a great deal of extra work when doctors process hundreds of patient messages every day. There is a risk that AI will not reduce the administrative burden, but will simply shift it to correcting automatically generated texts.
Personalised tailoring
At the same time, the research shows that AI does indeed have potential when the technology is better tailored to doctors’ individual communication styles. By training language models on a specific doctor’s previous responses, the quality of the generated messages improved by 33 per cent, whilst the number of corrections required fell by 26 per cent.
To achieve this, the researchers developed a new training method, TADPOLE (Thematic Agentic Direct Preference Optimisation for Learning Enhancement). This technique combines existing AI responses with messages actually written by doctors, enabling language models to better learn which content and tone are appropriate for clinical practice. According to the researchers, such an approach could ultimately lead to a significant reduction in the administrative burden. With higher-quality draft messages, doctors could potentially save one to two hours a day.
Empathy as an unexpected benefit
Although AI still regularly falls short in terms of content, the researchers also identified clear strengths. AI-generated responses tend to be more empathetic and comprehensive than messages written by doctors under severe time pressure. For instance, AI is more likely to begin by showing understanding for a patient’s situation and to provide clear instructions for any follow-up actions. According to the researchers, this characteristic could be harnessed to further improve patient communication.
AI can support doctors in formulating empathetic responses without taking over medical responsibility. The researchers therefore emphasise that, for the time being, AI should primarily be viewed as a supportive tool. Virtually no one expects patient messages to be handled entirely automatically without human oversight. The next step is therefore not full automation, but the development of systems that reduce the amount of correction work for doctors and better align with their clinical decision-making.
Further research must now determine how much time doctors actually save with improved AI models and how both healthcare professionals and patients perceive the quality of this support. According to the researchers, the healthcare sector has not yet reached the point where AI can independently take over patient communication, but the results do show that targeted optimisation can make the technology considerably more useful.
Earlier study
A study from Pennsylvania State University, conducted earlier this year, suggested that AI chatbots are not yet reliable enough to serve as standalone medical advisors. In a “Diagnose-a-thon,” 34 participants submitted 212 real-world health questions to ChatGPT, Gemini and Llama, after which nine board-certified physicians evaluated the responses. Overall, the AI systems produced medically accurate answers in 76.2% of cases, leaving a substantial risk of potentially harmful errors. Performance varied across specialties, with stronger results in obstetrics, gynecology and otolaryngology than in internal medicine, neurology and dermatology.
Researchers also found that concise, well-structured prompts generally produced more accurate responses. Surprisingly, enhancing the models with medical textbooks and clinical guidelines did not consistently improve their performance. The researchers conclude that, while AI has clear potential to support healthcare professionals, current chatbots should complement rather than replace clinical expertise and should not be relied on as independent sources of medical advice.
