Physicians beat symptom checkers in test of diagnostic accuracy

Irene Gabashvili irene at AURAMETRIX.COM
Wed Oct 12 17:47:07 UTC 2016


They found that computer algorithms were better at spotting less serious
conditions, while doctors were better at more uncommon diagnoses and more
serious conditions.

What computer systems are better performing, in your opinion and theirs?

On Wed, Oct 12, 2016 at 3:40 AM, Jason Maude <
jason.maude at isabelhealthcare.com> wrote:

> This is one of these studies where the purpose and limitations renders the
> study, at best, counter productive to the cause of improving diagnosis and,
> at worse, completely pointless.
>
>    1. What is the point of setting the study up as a competition between
>    the physician and symptom checker when that is not the purpose of a symptom
>    checker? Neither DDx generators or symptom checkers are credible or even
>    designed to replace a doctor and make the diagnosis but to make the doctor
>    and patient smarter. For our own long validation process we looked first at
>    the accuracy of the tool to see whether it was safe to use and then looked
>    at how it helped the doctor. Even though our internal studies show
>    comparable accuracy rates to the physicians in this study when using
>    several hundred cases from Medscape, we choose not to publish the data
>    precisely because it’s not a competition. You can see our validation
>    process from this link http://www.isabelhealthcare.com/
>    validation/peer-reviews
>    2. The symptom checker accuracy rates used in this study are simply
>    the average across 23 symptom checkers from the authors previous study. The
>    34% rate for getting the correct diagnosis is the average of a range that
>    went from 5% to 50% and the 51% rate for being in the top 3 is the average
>    of a range that went from 29% to 71%. How useful is it to compare
>    physicians to the average of a mixed bag of systems which are so completely
>    different?
>    3. As we pointed out when the original comparison of the 23 symptom
>    checkers appears, the test cases were medical clinical vignettes which
>    included cases with negative symptoms. Symptom checkers like Isabel cannot
>    understand negatives as use statistical natural language. How many patients
>    tell you about symptoms they don’t have?! To be fair, the authors do admit
>    that the test cases do not reflect the complexity of real word patients.
>    4. One significant result which did come out of the study was that the
>    Interns appeared more accurate overall than the Attendings (72 v 71.8 for
>    first and 89.5 v 82.7 for top 3)-that has some interesting implications!
>
> The authors should focus on how and whether these tools help the doctor
> and patient. It would be really helpful to understand how patients who have
> use these tools have been helped, how their doctor responded and whether
> they also found the consultation more productive. If these tools really do
> help the patient and doctor then we should vigorously encourage their use.
>
> Regards
> Jason
>
> Jason Maude
> Founder and CEO Isabel Healthcare
> Tel: +44 1428 644886
> Tel: +1 703 879 1890
> www.isabelhealthcare.com
>
>
> From: Peggy Zuckerman <peggyzuckerman at GMAIL.COM>
> Reply-To: Society to Improve Diagnosis in Medicine <IMPROVEDX at LIST.
> IMPROVEDIAGNOSIS.ORG>, Peggy Zuckerman <peggyzuckerman at GMAIL.COM>
> Date: Tuesday, 11 October 2016 22:01
> To: "IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG" <IMPROVEDX at LIST.
> IMPROVEDIAGNOSIS.ORG>
> Subject: Re: [IMPROVEDX] Physicians beat symptom checkers in test of
> diagnostic accuracy
>
> I have to assume that just as there are differences in the experience and
> quality of care given by doctors, that the same is true for symptom
> checkers.
>
> It would be interesting to know if the patient who checks his symptoms
> uses that knowledge to help in the diagnostic process.  If these symptom
> checkers/Dr. Google help the patient to learn the vocabulary so as to
> explain his symptoms better,  to know that one there are things to try at
> home before going to the doctor, or if, indeed, a doctor is needed, they
> are of value.
>
> No one should be surprised that doctors are better at this than patients
> and the symptom checkers, but should be supportive of the patients who are
> willing to learn about their own care options.
>
>
> Peggy Zuckerman
> www.peggyRCC.com
>
> On Tue, Oct 11, 2016 at 1:06 PM, HM Epstein <hmepstein at gmail.com> wrote:
>
>> Thank you so much for your analysis of the study. I agree that the closer
>> we can get to real world measurement of diagnostic successes and errors,
>> the better off we will be. It's interesting that the physicians were given
>> the same data that the symptom checkers were given and that they were
>> measurably more successful. Is that a problem with the symptom checkers or
>> the nature of a standardized patient case study?
>>
>> Best,
>> Helene
>>
>> *--  *
>>
>>
>> *Sent from my iPhone*
>>
>>
>>
>> On Oct 11, 2016, at 10:02 AM, Follansbee, William <follansbeewp at upmc.edu>
>> wrote:
>>
>> Helene,
>>
>>
>>
>> Thank you for sharing this article and I understand  you concern.  The
>> members following this listserv recognize the serious issue of diagnostic
>> error and are committed to the challenge of improving diagnostic accuracy.
>> At the same time, we need to remain objective in our interpretation of
>> data. The process of reaching a diagnosis is typically a longitudinal one
>> as patient symptoms evolve and as data are gathered over time. This study
>> measured diagnostic accuracy in a first impression type of scenario, with
>> no physical examination information nor any diagnostic testing information
>> such as laboratory data or radiologic studies. In that context, having the
>> correct diagnosis in the top three approximately 85% of the time may not be
>> as discouraging as it initially appears. Presumably as more information is
>> provided, clinical accuracy would improve.  On the other hand, this study
>> might also be overestimating diagnostic accuracy.  Standardized cases, as
>> were used in this study, is one approach which is attractive in the
>> research environment because it can be controlled and defined, but it is
>> less clear how well a study like this reflects real world experience.
>> Developing reliable methods to measure diagnostic error in the real world
>> is one of the major challenges we all face as we work to reduce its
>> frequency.
>>
>>
>>
>> Best regards,
>>
>>
>>
>> William P. Follansbee, M.D., FACC, FACP, FASNC, FAHA
>>
>> The Master Clinician Professor of Cardiovascular Medicine
>>
>> Director, The UPMC Clinical Center for Medical Decision Making
>>
>> Suite A429 UPMC Presbyterian
>>
>> 200 Lothrop Street
>>
>> Pittsburgh, PA 15213
>>
>> Phone: 412-647-3437
>>
>> Fax: 412-647-3873
>>
>> Email: follansbeewp at upmc.edu
>>
>>
>>
>> <image001.gif>
>>
>>
>>
>> *This email may contain confidential information of the sending
>> organization. Any unauthorized or improper disclosure, copying,
>> distribution, or use of the contents of this email and attached document(s)
>> is prohibited. The information contained in this email and attached
>> document(s) is intended only for the personal and confidential use of the
>> recipient(s) named above. If you have received this communication in error,
>> please notify the sender immediately by email and delete the original email
>> and attached document(s).*
>>
>>
>>
>>
>>
>>
>>
>> *From:* HM Epstein [mailto:hmepstein at GMAIL.COM <hmepstein at GMAIL.COM>]
>> *Sent:* Tuesday, October 11, 2016 12:18 AM
>> *To:* IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
>> <IMPROVEDX at list.improvediagnosis.org>
>> *Subject:* [IMPROVEDX] Physicians beat symptom checkers in test of
>> diagnostic accuracy
>>
>>
>>
>> Modern Healthcare's analysis of today's research letter in JAMA Internal
>> Medicine states that while participating physicians beat the symptom
>> checkers in identifying the correct Dx (which includes listing it in the
>> top three possible diagnoses), they were still off too often.
>>
>>
>>
>> "The physicians listed the correct diagnosis first across all cases 72.1%
>> of the time, while symptom checkers listed the correct diagnosis first only
>> 34% of the time, according to the research letter.
>>
>> Physicians also listed the correct diagnosis in their top three diagnoses
>> 84.3% of the time, while symptom checkers included the correct condition in
>> the top three 51.2% of the time."
>> http://www.modernhealthcare.com/article/20161010/NEWS/161019998
>>
>>
>>
>> As a patient, I'm depressed that physicians got the correct Dx less than
>> 3/4 of the time. It's more significant to me than the higher results
>> achieved for naming it in the top three possibilities, because patients
>> rarely hear the alternatives unless they specifically ask "What else could
>> it be?" To be fair, the lack of physical exams is a noteworthy element.
>>
>>
>>
>> Best,
>>
>> Helene
>>
>>
>>
>> *--  *
>>
>> *hmepstein.com <http://hmepstein.com> *
>>
>> *@hmepstein*
>>
>> *Mobile: 914-522-2116 <914-522-2116>*
>>
>>
>>
>> *Sent from my iPhone*
>>
>>
>>
>>
>>
>>
>> ------------------------------
>>
>>
>
> ------------------------------
>
>
> To unsubscribe from IMPROVEDX: click the following link:
> http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?
> SUBED1=IMPROVEDX&A=1 or send email to: IMPROVEDX-SIGNOFF-REQUEST@
> LIST.IMPROVEDIAGNOSIS.ORG
>
> Moderator:David Meyers, Board Member, Society for Improving Diagnosis in
> Medicine
>
> To learn more about SIDM visit:
> http://www.improvediagnosis.org/
>






Moderator: David Meyers, Board Member, Society to Improve Diagnosis in Medicine


HTML Version:
URL: <../attachments/20161012/af11f90b/attachment.html>


More information about the Test mailing list