- 78.6% of medical experts preferred ChatGPT’s answers to those of a physician, per a new study.
- Experts found the chatbot’s responses to patient questions were higher quality and more empathetic.
- ChatGPT can still make grave medical errors, but this study suggests AI may improve upon a doctor’s bedside manner.
ChatGPT may be just as good as doctors — if not better — at responding to patients’ medical questions, according to a new study.
Researchers from the University of California, San Diego; John Hopkins; and other universities asked OpenAI’s ChatGPT 195 questions posted to Reddit’s AskDocs forum, and compared both the quality and the empathy of the chatbot’s responses to answers from verified physicians on Reddit.
A team of healthcare experts, including professionals working in internal medicine, pediatrics, oncology, and infectious disease, scored both the bot and the human answers on a five-point scale, assessing the “quality of information” and “the empathy or bedside manner” provided.
In the study, clinicians preferred the chatbot’s response to the physician’s in 78.6% of the 585 scenarios. The chatbot’s responses were rated 3.6-times higher in quality and 9.8-times higher in empathy than those of the physicians.
Doctors keep it brief — while ChatGPT responds at length
A big reason ChatGPT won out in the study is because the bot’s responses to questions were longer and more personable than the physicians’ brief, time-saver answers.
For example, when asked whether it’s possible to go blind after getting bleach in your eye, ChatGPT responded “I’m sorry to hear that you got bleach splashed in your eye,” and offered four additional sentences of explanation, with specific instructions on how to clean it.
The physician just said, “Sounds like you will be fine,” and briefly advised the patient to “flush the eye” or call poison control.
Another patient’s question about a “hard lump under the skin of the penis” yielded a “not an emergency” response from the doctor, who then advised the patient to make an appointment for the “first available slot” with their physician or urologist. ChatGPT, on the other hand, offered a far more explanatory three-paragraph response detailing the possibility of penile cancer.
“It is important to remember that the vast majority of lumps or bumps on the penis are benign and are not cancerous,” the chatbot said. “However, it is important to have any new or unusual lump or bump evaluated.”
Similar responses were generated for treating a lingering cough, advice on going to the emergency room following a blow to the head, and concerns about finding traces of blood in feces.
ChatGPT isn’t really capable of diagnosing on its own
Don’t let ChatGPT’s performance in this study fool you. It’s still not a doctor.
