Physicians beat symptom checkers in test of diagnostic accuracy

Jason Maude jason.maude at ISABELHEALTHCARE.COM
Wed Oct 12 10:40:52 UTC 2016

This is one of these studies where the purpose and limitations renders the study, at best, counter productive to the cause of improving diagnosis and, at worse, completely pointless.

  1.  What is the point of setting the study up as a competition between the physician and symptom checker when that is not the purpose of a symptom checker? Neither DDx generators or symptom checkers are credible or even designed to replace a doctor and make the diagnosis but to make the doctor and patient smarter. For our own long validation process we looked first at the accuracy of the tool to see whether it was safe to use and then looked at how it helped the doctor. Even though our internal studies show comparable accuracy rates to the physicians in this study when using several hundred cases from Medscape, we choose not to publish the data precisely because it’s not a competition. You can see our validation process from this link
  2.  The symptom checker accuracy rates used in this study are simply the average across 23 symptom checkers from the authors previous study. The 34% rate for getting the correct diagnosis is the average of a range that went from 5% to 50% and the 51% rate for being in the top 3 is the average of a range that went from 29% to 71%. How useful is it to compare physicians to the average of a mixed bag of systems which are so completely different?
  3.  As we pointed out when the original comparison of the 23 symptom checkers appears, the test cases were medical clinical vignettes which included cases with negative symptoms. Symptom checkers like Isabel cannot understand negatives as use statistical natural language. How many patients tell you about symptoms they don’t have?! To be fair, the authors do admit that the test cases do not reflect the complexity of real word patients.
  4.  One significant result which did come out of the study was that the Interns appeared more accurate overall than the Attendings (72 v 71.8 for first and 89.5 v 82.7 for top 3)-that has some interesting implications!

The authors should focus on how and whether these tools help the doctor and patient. It would be really helpful to understand how patients who have use these tools have been helped, how their doctor responded and whether they also found the consultation more productive. If these tools really do help the patient and doctor then we should vigorously encourage their use.


Jason Maude
Founder and CEO Isabel Healthcare
Tel: +44 1428 644886
Tel: +1 703 879 1890<>

From: Peggy Zuckerman <peggyzuckerman at GMAIL.COM<mailto:peggyzuckerman at GMAIL.COM>>
Reply-To: Society to Improve Diagnosis in Medicine <IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>>, Peggy Zuckerman <peggyzuckerman at GMAIL.COM<mailto:peggyzuckerman at GMAIL.COM>>
Date: Tuesday, 11 October 2016 22:01
Subject: Re: [IMPROVEDX] Physicians beat symptom checkers in test of diagnostic accuracy

I have to assume that just as there are differences in the experience and quality of care given by doctors, that the same is true for symptom checkers.

It would be interesting to know if the patient who checks his symptoms uses that knowledge to help in the diagnostic process.  If these symptom checkers/Dr. Google help the patient to learn the vocabulary so as to explain his symptoms better,  to know that one there are things to try at home before going to the doctor, or if, indeed, a doctor is needed, they are of value.

No one should be surprised that doctors are better at this than patients and the symptom checkers, but should be supportive of the patients who are willing to learn about their own care options.

Peggy Zuckerman<>

On Tue, Oct 11, 2016 at 1:06 PM, HM Epstein <hmepstein at<mailto:hmepstein at>> wrote:
Thank you so much for your analysis of the study. I agree that the closer we can get to real world measurement of diagnostic successes and errors, the better off we will be. It's interesting that the physicians were given the same data that the symptom checkers were given and that they were measurably more successful. Is that a problem with the symptom checkers or the nature of a standardized patient case study?


Sent from my iPhone

On Oct 11, 2016, at 10:02 AM, Follansbee, William <follansbeewp at<mailto:follansbeewp at>> wrote:


Thank you for sharing this article and I understand  you concern.  The members following this listserv recognize the serious issue of diagnostic error and are committed to the challenge of improving diagnostic accuracy. At the same time, we need to remain objective in our interpretation of data. The process of reaching a diagnosis is typically a longitudinal one as patient symptoms evolve and as data are gathered over time. This study measured diagnostic accuracy in a first impression type of scenario, with no physical examination information nor any diagnostic testing information such as laboratory data or radiologic studies. In that context, having the correct diagnosis in the top three approximately 85% of the time may not be as discouraging as it initially appears. Presumably as more information is provided, clinical accuracy would improve.  On the other hand, this study might also be overestimating diagnostic accuracy.  Standardized cases, as were used in this study, is one approach which is attractive in the research environment because it can be controlled and defined, but it is less clear how well a study like this reflects real world experience. Developing reliable methods to measure diagnostic error in the real world is one of the major challenges we all face as we work to reduce its frequency.

Best regards,

William P. Follansbee, M.D., FACC, FACP, FASNC, FAHA
The Master Clinician Professor of Cardiovascular Medicine
Director, The UPMC Clinical Center for Medical Decision Making
Suite A429 UPMC Presbyterian
200 Lothrop Street
Pittsburgh, PA 15213
Phone: 412-647-3437<tel:412-647-3437>
Fax: 412-647-3873<tel:412-647-3873>
Email: follansbeewp at<mailto:follansbeewp at>


This email may contain confidential information of the sending organization. Any unauthorized or improper disclosure, copying, distribution, or use of the contents of this email and attached document(s) is prohibited. The information contained in this email and attached document(s) is intended only for the personal and confidential use of the recipient(s) named above. If you have received this communication in error, please notify the sender immediately by email and delete the original email and attached document(s).

From: HM Epstein [mailto:hmepstein at GMAIL.COM]
Sent: Tuesday, October 11, 2016 12:18 AM
Subject: [IMPROVEDX] Physicians beat symptom checkers in test of diagnostic accuracy

Modern Healthcare's analysis of today's research letter in JAMA Internal Medicine states that while participating physicians beat the symptom checkers in identifying the correct Dx (which includes listing it in the top three possible diagnoses), they were still off too often.

"The physicians listed the correct diagnosis first across all cases 72.1% of the time, while symptom checkers listed the correct diagnosis first only 34% of the time, according to the research letter.

Physicians also listed the correct diagnosis in their top three diagnoses 84.3% of the time, while symptom checkers included the correct condition in the top three 51.2% of the time."

As a patient, I'm depressed that physicians got the correct Dx less than 3/4 of the time. It's more significant to me than the higher results achieved for naming it in the top three possibilities, because patients rarely hear the alternatives unless they specifically ask "What else could it be?" To be fair, the lack of physical exams is a noteworthy element.

Mobile: 914-522-2116<tel:914-522-2116>

Sent from my iPhone


To unsubscribe from the IMPROVEDX:
or click the following link: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG


For additional information and subscription commands, visit:

http://LIST.IMPROVEDIAGNOSIS.ORG/ (with your password)

Visit the searchable archives or adjust your subscription at:

Moderator: David Meyers, Board Member, Society to Improve Diagnosis in Medicine

To unsubscribe from the IMPROVEDX list, click the following link:<br>
<a href="" target="_blank"></a>

HTML Version:
URL: <../attachments/20161012/bceb80d5/attachment.html>

More information about the Test mailing list