Physicians beat symptom checkers in test of diagnostic accuracy

Jason Maude jason.maude at ISABELHEALTHCARE.COM
Fri Oct 14 14:38:16 UTC 2016


Shantanu
When the press write about DDx tools or symptom checkers they often hang their story around whether they are better or worse than physicians and whether physicians will eventually be replaced by them. This is unhelpful and potentially damaging to the cause of trying to encourage the adoption of these tools but I can understand how the press goes for this angle to spice up the story. Apart from the specific comments, my main issue with this study is that it appears to add credence to this line of argument in a medical journal when it’s counter productive.

  1.  To your first point, I am not sure that was a serious limitation and since your study shows that the Interns did better than the Attendings what does that tell us about the nature of the cases used? The more serious limitation was that the symptom checkers were tested with clinical vignettes which are very different to the way actual patients describe and enter their cases.
  2.  To your second point around the accuracy of the symptom checkers, you have used an average across a very wide range of very different systems in addition to testing them in a very different way to how patients would actually use them. The headline accuracy rates used of 34% and 51% are, therefore, quite misleading.
  3.  Patients using symptom checkers in lieu of seeing a doctor: I am sure some do but the overwhelming body of feedback we receive shows that patients don’t and the vast majority are using them just to get better informed and understand their symptoms better - they seem to get a sense of relief having used them. You can see some of the comments we have received from this link http://www.isabelhealthcare.com/customer-satisfaction/testimonials-patients
  4.  The results of the study were not obvious because a/ the symptom checkers designed for patients were tested with medical cases b/ the overall accuracy figures used were misleading averages and c/ the fact the Interns performed better than the Attendings raises questions around the nature of the cases used.

Regards
Jason

Jason Maude
Founder and CEO Isabel Healthcare
Tel: +44 1428 644886
Tel: +1 703 879 1890
www.isabelhealthcare.com<http://www.isabelhealthcare.com/>

From: Shantanu Nundy <shantanu at HUMANDX.ORG<mailto:shantanu at HUMANDX.ORG>>
Reply-To: Society to Improve Diagnosis in Medicine <IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>>, Shantanu Nundy <shantanu at HUMANDX.ORG<mailto:shantanu at HUMANDX.ORG>>
Date: Thursday, 13 October 2016 17:20
To: "IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>" <IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>>
Subject: Re: [IMPROVEDX] Physicians beat symptom checkers in test of diagnostic accuracy

Hi Colleagues,
One of the authors of the study, I want to thank you for sharing our work with the SIDM community and for your comments.

A few thoughts below:
- One of the primary aims of the study was to address a major limitation of the original Harvard study on symptom checkers- that the performance of physicians on these same vignettes was not known and thus the accuracy/validity/difficulty of these cases could not be assessed.

- In terms of implications, the use case here isn't a doctor making a diagnosis her-/himself vs a doctor using a symptom checker. It's from the patient perspective. Patients sometimes use symptom checkers in lieu of a medical visit. Thus comparing the accuracy of physicians vs symptom checkers is relevant. I agree that the clinical decision support use case is entirely different.

- The primary result wasn't shocking but at the same time was not obvious. What's important to understand is how simple these cases are- they contain very limited information, are incomplete, and are classic presentations. Thus it is interesting that physicians performed so much better on them than symptom checkers. This says something about the ability of physicians to reason on sparse data.

Finally I'd caution us as a community to be so abrasive in our discussion posts. We are all in this. Together.

Shantanu


On Oct 13, 2016, at 12:09 AM, Peggy Zuckerman <peggyzuckerman at GMAIL.COM<mailto:peggyzuckerman at GMAIL.COM>> wrote:

How hard is to think that the exquisitely complex mind would benefit from a huge filing system, with lots of search options and the ability to ask in digita talk, "What else could it be?"

Peggy Zuckerman

Peggy Zuckerman
www.peggyRCC.com<http://www.peggyRCC.com>

On Wed, Oct 12, 2016 at 2:26 PM, Art Papier <apapier at visualdx.com<mailto:apapier at visualdx.com>> wrote:
I agree with Ed and Jason and will make it n=3.  Charles Friedman PhD has a an equation for describing professional clinical decision support:

brain + computer > brain

Users of professional diagnostic decision support know the value of CDS and live this equation, and improve care with professional diagnostic CDS.  I am wondering if the physicians on the listserv know of colleagues relying on WebMD or similar for differential diagnosis?

I do not know any clinicians using WebMD for diagnosis, and really don’t understand the premise of this comparison as Ed and Jason communicate.


Art Papier MD
CEO
phone 585-272-2630<tel:585-272-2630>
mobile 585-615-8245<tel:585-615-8245>
address 339 East Ave, Suite 410 Rochester, NY 14604
web visualdx.com<http://visualdx.com/>
email apapier at visualdx.com<mailto:apapier at visualdx.com>

<image001.jpg><http://facebook.com/visualdx>  <image002.jpg><http://www.twitter.com/visualdx>  [https://newoldstamp.com/editor/images/in.jpg] <http://linkedin.com/company/logical-images>   <image004.jpg><http://instagram.com/visualdximages>

<image005.png><http://www.visualdx.com/>


[cid:image006.jpg at 01D159E6.3E7218B0]



From: Hoffer, Edward P.,M.D. [mailto:EHOFFER at MGH.HARVARD.EDU<mailto:EHOFFER at MGH.HARVARD.EDU>]
Sent: Wednesday, October 12, 2016 9:43 AM

To: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>
Subject: Re: [IMPROVEDX] Physicians beat symptom checkers in test of diagnostic accuracy

Excellent point. “Symptom checkers” as used by medical professionals are intended to assist them, not replace them.
A much better study, which we and others have done, is to compare the accuracy of diagnosis with and without the use of a diagnostic decision support system such as Isabel or DXplain. These studies generally show improved diagnostic accuracy when the DDSS is used by the clinician, particularly if used early in the diagnostic process.
Edward Hoffer MD, FACC, FACMI
From: Jason Maude [mailto:jason.maude at ISABELHEALTHCARE.COM]
Sent: Wednesday, October 12, 2016 6:41 AM
To: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>
Subject: Re: [IMPROVEDX] Physicians beat symptom checkers in test of diagnostic accuracy

This is one of these studies where the purpose and limitations renders the study, at best, counter productive to the cause of improving diagnosis and, at worse, completely pointless.

  1.  What is the point of setting the study up as a competition between the physician and symptom checker when that is not the purpose of a symptom checker? Neither DDx generators or symptom checkers are credible or even designed to replace a doctor and make the diagnosis but to make the doctor and patient smarter. For our own long validation process we looked first at the accuracy of the tool to see whether it was safe to use and then looked at how it helped the doctor. Even though our internal studies show comparable accuracy rates to the physicians in this study when using several hundred cases from Medscape, we choose not to publish the data precisely because it’s not a competition. You can see our validation process from this link http://www.isabelhealthcare.com/validation/peer-reviews
  2.  The symptom checker accuracy rates used in this study are simply the average across 23 symptom checkers from the authors previous study. The 34% rate for getting the correct diagnosis is the average of a range that went from 5% to 50% and the 51% rate for being in the top 3 is the average of a range that went from 29% to 71%. How useful is it to compare physicians to the average of a mixed bag of systems which are so completely different?
  3.  As we pointed out when the original comparison of the 23 symptom checkers appears, the test cases were medical clinical vignettes which included cases with negative symptoms. Symptom checkers like Isabel cannot understand negatives as use statistical natural language. How many patients tell you about symptoms they don’t have?! To be fair, the authors do admit that the test cases do not reflect the complexity of real word patients.
  4.  One significant result which did come out of the study was that the Interns appeared more accurate overall than the Attendings (72 v 71.8 for first and 89.5 v 82.7 for top 3)-that has some interesting implications!
The authors should focus on how and whether these tools help the doctor and patient. It would be really helpful to understand how patients who have use these tools have been helped, how their doctor responded and whether they also found the consultation more productive. If these tools really do help the patient and doctor then we should vigorously encourage their use.

Regards
Jason

Jason Maude
Founder and CEO Isabel Healthcare
Tel: +44 1428 644886<tel:%2B44%201428%20644886>
Tel: +1 703 879 1890<tel:%2B1%20703%20879%201890>
www.isabelhealthcare.com<http://www.isabelhealthcare.com/>


From: Peggy Zuckerman <peggyzuckerman at GMAIL.COM<mailto:peggyzuckerman at GMAIL.COM>>
Reply-To: Society to Improve Diagnosis in Medicine <IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>>, Peggy Zuckerman <peggyzuckerman at GMAIL.COM<mailto:peggyzuckerman at GMAIL.COM>>
Date: Tuesday, 11 October 2016 22:01
To: "IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>" <IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>>
Subject: Re: [IMPROVEDX] Physicians beat symptom checkers in test of diagnostic accuracy

I have to assume that just as there are differences in the experience and quality of care given by doctors, that the same is true for symptom checkers.

It would be interesting to know if the patient who checks his symptoms uses that knowledge to help in the diagnostic process.  If these symptom checkers/Dr. Google help the patient to learn the vocabulary so as to explain his symptoms better,  to know that one there are things to try at home before going to the doctor, or if, indeed, a doctor is needed, they are of value.

No one should be surprised that doctors are better at this than patients and the symptom checkers, but should be supportive of the patients who are willing to learn about their own care options.


Peggy Zuckerman
www.peggyRCC.com<http://www.peggyRCC.com>

On Tue, Oct 11, 2016 at 1:06 PM, HM Epstein <hmepstein at gmail.com<mailto:hmepstein at gmail.com>> wrote:
Thank you so much for your analysis of the study. I agree that the closer we can get to real world measurement of diagnostic successes and errors, the better off we will be. It's interesting that the physicians were given the same data that the symptom checkers were given and that they were measurably more successful. Is that a problem with the symptom checkers or the nature of a standardized patient case study?

Best,
Helene
--


Sent from my iPhone



On Oct 11, 2016, at 10:02 AM, Follansbee, William <follansbeewp at upmc.edu<mailto:follansbeewp at upmc.edu>> wrote:
Helene,

Thank you for sharing this article and I understand  you concern.  The members following this listserv recognize the serious issue of diagnostic error and are committed to the challenge of improving diagnostic accuracy. At the same time, we need to remain objective in our interpretation of data. The process of reaching a diagnosis is typically a longitudinal one as patient symptoms evolve and as data are gathered over time. This study measured diagnostic accuracy in a first impression type of scenario, with no physical examination information nor any diagnostic testing information such as laboratory data or radiologic studies. In that context, having the correct diagnosis in the top three approximately 85% of the time may not be as discouraging as it initially appears. Presumably as more information is provided, clinical accuracy would improve.  On the other hand, this study might also be overestimating diagnostic accuracy.  Standardized cases, as were used in this study, is one approach which is attractive in the research environment because it can be controlled and defined, but it is less clear how well a study like this reflects real world experience. Developing reliable methods to measure diagnostic error in the real world is one of the major challenges we all face as we work to reduce its frequency.

Best regards,

William P. Follansbee, M.D., FACC, FACP, FASNC, FAHA
The Master Clinician Professor of Cardiovascular Medicine
Director, The UPMC Clinical Center for Medical Decision Making
Suite A429 UPMC Presbyterian
200 Lothrop Street
Pittsburgh, PA 15213
Phone: 412-647-3437<tel:412-647-3437>
Fax: 412-647-3873<tel:412-647-3873>
Email: follansbeewp at upmc.edu<mailto:follansbeewp at upmc.edu>

<image001.gif>

This email may contain confidential information of the sending organization. Any unauthorized or improper disclosure, copying, distribution, or use of the contents of this email and attached document(s) is prohibited. The information contained in this email and attached document(s) is intended only for the personal and confidential use of the recipient(s) named above. If you have received this communication in error, please notify the sender immediately by email and delete the original email and attached document(s).



From: HM Epstein [mailto:hmepstein at GMAIL.COM]
Sent: Tuesday, October 11, 2016 12:18 AM
To: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at list.improvediagnosis.org>
Subject: [IMPROVEDX] Physicians beat symptom checkers in test of diagnostic accuracy

Modern Healthcare's analysis of today's research letter in JAMA Internal Medicine states that while participating physicians beat the symptom checkers in identifying the correct Dx (which includes listing it in the top three possible diagnoses), they were still off too often.

"The physicians listed the correct diagnosis first across all cases 72.1% of the time, while symptom checkers listed the correct diagnosis first only 34% of the time, according to the research letter.

Physicians also listed the correct diagnosis in their top three diagnoses 84.3% of the time, while symptom checkers included the correct condition in the top three 51.2% of the time."
http://www.modernhealthcare.com/article/20161010/NEWS/161019998

As a patient, I'm depressed that physicians got the correct Dx less than 3/4 of the time. It's more significant to me than the higher results achieved for naming it in the top three possibilities, because patients rarely hear the alternatives unless they specifically ask "What else could it be?" To be fair, the lack of physical exams is a noteworthy element.

Best,
Helene
--
hmepstein.com<http://hmepstein.com>
@hmepstein
Mobile: 914-522-2116<tel:914-522-2116>

Sent from my iPhone



________________________________


________________________________

Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>

To unsubscribe from IMPROVEDX: click the following link:
http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1
or send email to: IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG>

Visit the searchable archives or adjust your subscription at: http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?INDEX

Moderator:David Meyers, Board Member, Society for Improving Diagnosis in Medicine

To learn more about SIDM visit:
http://www.improvediagnosis.org/

The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

________________________________

Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>

To unsubscribe from IMPROVEDX: click the following link:
http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1
or send email to: IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG>

Visit the searchable archives or adjust your subscription at: http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?INDEX

Moderator:David Meyers, Board Member, Society for Improving Diagnosis in Medicine

To learn more about SIDM visit:
http://www.improvediagnosis.org/

________________________________

Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>

To unsubscribe from IMPROVEDX: click the following link:
http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1 or send email to: IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG>

Visit the searchable archives or adjust your subscription at: http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?INDEX
Moderator:David Meyers, Board Member, Society for Improving Diagnosis in Medicine

To learn more about SIDM visit:
http://www.improvediagnosis.org/


________________________________

Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>

To unsubscribe from IMPROVEDX: click the following link:
http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1

or send email to: IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG>

Visit the searchable archives or adjust your subscription at: http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?INDEX

Moderator:David Meyers, Board Member, Society for Improving Diagnosis in Medicine

To learn more about SIDM visit:
http://www.improvediagnosis.org/

________________________________

Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>

To unsubscribe from IMPROVEDX: click the following link:
http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1

or send email to: IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG>

Visit the searchable archives or adjust your subscription at: http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?INDEX

Moderator:David Meyers, Board Member, Society for Improving Diagnosis in Medicine

To learn more about SIDM visit:
http://www.improvediagnosis.org/

To unsubscribe from the IMPROVEDX:
mail to:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG
or click the following link: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG

Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG

For additional information and subscription commands, visit:
http://www.lsoft.com/resources/faq.asp#4A

http://LIST.IMPROVEDIAGNOSIS.ORG/ (with your password)

Visit the searchable archives or adjust your subscription at:
http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?INDEX

Moderator: David Meyers, Board Member, Society to Improve Diagnosis in Medicine

To unsubscribe from the IMPROVEDX list, click the following link:<br>
<a href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1" target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1</a>
</p>

HTML Version:
URL: <../attachments/20161014/ec51db2b/attachment.html> ATTACHMENT:
Name: image003.jpg Type: image/jpeg Size: 778 bytes Desc: image003.jpg URL: <../attachments/20161014/ec51db2b/attachment.jpg> ATTACHMENT:
Name: image006.jpg Type: image/jpeg Size: 4137 bytes Desc: image006.jpg URL: <../attachments/20161014/ec51db2b/attachment-0001.jpg>


More information about the Test mailing list