Can Doctors Learn Something from Chatbots?

Educational Objectives

Summary

Can Doctors Learn Something from Chatbots?

In some contexts, artificial intelligence (AI) chatbots can communicate as well as or even better than humans. If medicine proves to be one of them, the technology might help with some routine physician duties, like responding to patients’ emails and texts. For a study on the website of JAMA Internal Medicine (https://doi.org/10.1001/jamainternmed.2023.1838), researchers selected about 200 anonymous medical questions posted by members of the public in 2022 to a Reddit website (https://www.reddit.com/r/AskDocs/), which were answered online by anonymous volunteer physicians with verified credentials. The same questions were then posed verbatim to ChatGPT. A team of physicians compared the responses in a blinded fashion.

Overall, the evaluators preferred the chatbot responses almost 80% of the time. The chatbot’s responses were longer (mean, 180 vs 52 words) and ranked higher in quality of information (78% of chatbot responses were categorized as “good” or “very good” as opposed to 22% of physicians’ responses). Chatbot responses also were far kinder (45% of the bot’s responses were “empathetic” or “very empathetic,” as opposed to 5% of physicians’ responses). A table comparing a few sample answers confirms that physicians tended to be curt and often dismissive, whereas bots were pleasant and concerned.

This study has significant limitations. It says nothing about how doctors interact with their own patients and nothing about how patients themselves might react to the two sets of answers. Perhaps the Reddit physicians thought their job was to be as brief and unemotional as possible. Even so, the bottom line is clear: Chatbots can be trained to be superb medical communicators. So here’s the real question, at least for me: Can’t humans be trained, too? All healthcare personnel might study some of these canned answers for tips before we decide just to pass the baton to the bots.

Abigail Zuger, MD

Readings

Disclosures

Dr. Christopher Longhurst reported being an equity holder in Doximity as a former advisor.

Acknowledgements

CME/CE INFO

Accreditation:

The Audio- Digest Foundation is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians.

The Audio- Digest Foundation designates this enduring material for a maximum of 0.00 AMA PRA Category 1 Credits™. Physicians should claim only the credit commensurate with the extent of their participation in the activity.

Audio Digest Foundation is accredited as a provider of continuing nursing education by the American Nurses Credentialing Center's (ANCC's) Commission on Accreditation. Audio Digest Foundation designates this activity for 0.00 CE contact hours.

Lecture ID:

JW341111

Expiration:

This CME course qualifies for AMA PRA Category 1 Credits™ for 3 years from the date of publication.

Instructions:

To earn CME/CE credit for this course, you must complete all the following components in the order recommended: (1) Review introductory course content, including Educational Objectives and Faculty/Planner Disclosures; (2) Listen to the audio program and review accompanying learning materials; (3) Complete posttest (only after completing Step 2) and earn a passing score of at least 80%. Taking the course Pretest and completing the Evaluation Survey are strongly recommended (but not mandatory) components of completing this CME/CE course.

Estimated time to complete this CME/CE course:

Approximately 2x the length of the recorded lecture to account for time spent studying accompanying learning materials and completing tests.

More Details - Certification & Accreditation