ChatGPT responses to patients rated more empathetic than doctors – research

ChatGPT also appeared to score higher than doctors on the quality of responses to patients.
Researchers said further studies are needed to evaluate whether chatbots such as ChatGPT can be used in clinical settings to help reduce burnout in doctors (John Walton/PA)
PA Wire
Nilima Marshall28 April 2023

A chatbot appears to be “significantly more empathetic” than doctors when responding to questions from patients, according to scientists.

Researchers asked a team of licensed healthcare professionals to rate responses from doctors as well as ChatGPT, a computer program designed to simulate online conversations with humans.

They found that the proportion of responses rated “empathetic” or “very empathetic” was higher for ChatGPT than for physicians.

ChatGPT also appeared to score higher than doctors on the quality of responses to patients.

Further exploration of this technology is warranted in clinical settings, such as using chatbot to draft responses that physicians could then edit

Study authors

Writing in the journal Jama Internal Medicine, the researchers said further studies are needed to evaluate whether chatbots such as ChatGPT can be used in clinical settings to help reduce burnout in doctors and other healthcare professionals.

They said: “In this cross-sectional study, a chatbot generated quality and empathetic responses to patient questions posed in an online forum.

“Further exploration of this technology is warranted in clinical settings, such as using chatbot to draft responses that physicians could then edit.

“Randomised trials could assess further if using AI assistants might improve responses, lower clinician burnout, and improve patient outcomes.”

For the study, the researchers looked at the questions asked by patients on the social media forum Reddit, which were answered by a verified physician.

The same questions were then put forward to ChatGPT.

Responses from doctors and ChatGPT were anonymised and randomly rated by healthcare professionals.

The human relationship and care-giving is a vital part of the healthcare process and something which cannot be automated or replaced by chatbots such as ChatGPT

Dr Mhairi Aitken, The Alan Turing Institute

Of the 195 questions and responses, results showed that, overall, evaluators preferred the chatbot’s responses to the doctors’.

ChatGPT responses were also rated significantly more empathetic than physician responses, the researchers said, and the proportion of responses rated as “good” or “very good” quality was higher for the chatbot than for physicians.

Commenting on the study, Mirella Lapata, professor of natural language processing, University of Edinburgh, said: “The study assesses ChatGPT’s ability to provide responses to patient questions and compares these to answers written by physicians.

She added: “Without controlling for the length of the response, we cannot know for sure whether the raters judged for style (eg, verbose and flowery discourse) rather than content.”

Dr Mhairi Aitken, ethics research fellow at The Alan Turing Institute, said that it is important to consider the perspectives of patients and not just professionals when using chatbots.

She added: “It’s important to note that while some people may feel comfortable receiving medical advice from a chatbot, or for a chatbot to assist in a physician’s advice, for many patients the human relationship and care-giving is a vital part of the healthcare process and something which cannot be automated or replaced by chatbots such as ChatGPT.

“A human doctor is able to adjust their language, manner, and approach in response to social cues and interactions, whereas a chatbot will produce more generic language without awareness of social contexts.”

Create a FREE account to continue reading

eros

Registration is a free and easy way to support our journalism.

Join our community where you can: comment on stories; sign up to newsletters; enter competitions and access content on our app.

Your email address

Must be at least 6 characters, include an upper and lower case character and a number

You must be at least 18 years old to create an account

* Required fields

Already have an account? SIGN IN

By clicking Create Account you confirm that your data has been entered correctly and you have read and agree to our Terms of use , Cookie policy and Privacy policy .

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Thank you for registering

Please refresh the page or navigate to another page on the site to be automatically logged in