Error Rates: Diagnosis--neither numerator nor denominator is known

Robert L Wears, MD, MS, PhD wears at UFL.EDU
Mon Dec 16 21:23:12 UTC 2013


It's even worse than that.

Suppose the treatment removes the evidence that it was even needed, so in retrospect it 
looks like unnecessary treatment?  Aviation has encountered this problem in pilots' 
decisionmaking; fundamentally, it is an issue in making a judgment about the process based 
on the outcome, confusing antecendents and consequents.

bob


On 16 Dec 2013 at 15:59, Jackson, Brian wrote:

> Suppose a physician judges that a patient has a 10% chance of a
> serious, treatable condition, and that initiating treatment
> immediately outweighs the risks.  And then suppose that the patient
> turns out not to have that particular condition.  Was the initial
> action based on a misdiagnosis?  Or would the opposite action, namely
> withholding treatment, have been considered an error?  Now imagine
> that there's a clinical practice guideline that explicitly recommends
> immediate treatment for this condition provided that the probability
> is judged to be at least 10%.  That seems to put the physician on
> solid ground, right?  But now imagine that the patient died from the
> side effect of the treatment, and on retrospective review (M&M?) a
> different expert physician judges that the patient had only a 5% a
> priori probability of that condition.  Now, was it an error?
> 
> (Some readers might want to cop out by calling this a question of
> therapeutic error, but the assumption here is that the treatment
> decision follows directly from the diagnostic assessment.) 
> 
> Given that medical diagnosis deals in probabilities rather than
> absolutes, and that many cases have considerable ambiguity, I'm
> concerned about the potential consequences of labeling specific
> incidents as errors.  The legal industry boils things down to
> absolutes, and we've seen how that works.  Our goal is to avoid
> errors, but maybe we can't measure individual errors directly, and it
> may even be counterproductive to even try to do so.  Indirect
> (process) measurement may well be more practical for most situations. 
> And outcome measures might be best based on estimation over large data
> sets.  In principle, any of these measures could be framed either
> positively (diagnostic success measures) or negatively (error
> measures) but based on Dr. Centor's suggestion, use of "error"
> terminology might be better reserved for use in the abstract, and
> "success" terminology for specific, labeled settings.
> 
> ..........................................
> Brian R. Jackson, MD, MS
> VP - Chief Medical Informatics Officer, ARUP Laboratories
> Assoc. Professor of Pathology (Clinical), University of Utah
> 
> 500 Chipeta Way, Mail Code 100
> Salt Lake City, Utah 84108-1221
> phone: (801) 583-2787, extension 1-3191
> toll free: (800) 242-2787
> fax: (801)584-5108
> email: brian.jackson at aruplab.com
> web: www.aruplab.com
> 
> 
> 
> -----Original Message-----
> From: Robert M Centor [mailto:rcentor at UAB.EDU] 
> Sent: Monday, December 16, 2013 7:16 AM
> To: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
> Subject: Re: [IMPROVEDX] Error Rates: Diagnosis--neither numerator nor
> denominator is known
> 
> While diagnostic errors are both the name of the meeting and the
> impetus of the society, we should not ignore the road to diagnostic
> expertise. Many diagnostic errors are really the mirror image of
> diagnostic expertise.  We must help physicians become better
> diagnosticians, not by berating them for errors, but by developing and
> explaining the path to diagnostic success.  Pronovost succeeded
> because he showed others how to have central lines without infections.
>  Of course he taught them to avoid infections, but his work appeals to
> everyone because of success, not just the absence of failure.
> ==============
> 
> Robert M Centor, MD, FACP
> 
> Regional Dean, UAB Huntsville Regional Medical Campus
> 301 Governors Drive
> Huntsville, AL 35801
> 
> Office: 256-539-7757
> Fax: 256-551-4451
> 
> Chair-Elect, ACP Board of Regents
> 
> Professor, General Internal Medicine
> UAB
> FOT 720
> 1530 3rd Ave S
> Birmingham, AL 35294-3407
> Office: 205-975-4889
> 
> 
> 
> 
> 
> On 12/15/13, 10:32 PM, "Swerlick, Robert A" <rswerli at EMORY.EDU> wrote:
> 
> >While the issue of diagnostic error is the central focus of this
> >organization, we continue to stumble when it comes to understanding
> >frequency and definitions. We all know it exists and can identify
> >specific examples within our own worlds but are hard pressed to come
> >up with great approaches to measure.
> >
> >One of the strengths of the SIDM is the diversity of participants. It
> > is also one of the weaknesses because the nature is diagnostic
> >errors observed and/or experienced varies hugely across the spectrum
> >of SIDM members. The is the inpatient/outpatient divide, the acute
> >care/chronic care divide, urgent/non-urgent divide, and
> >practitioner/patient divide. I also think there is a divide regarding
> >under-diagnosis (missed diagnosis) vs. over-diagnosis often playing
> >out in populations of patients who might be viewed as well (no
> >symptoms).
> >
> >The priorities of these different constituencies may be very
> >different and we may be best served by recognizing up front that the
> >challenges and priorities and tools used to get at diagnostic errors
> >in these scenarios will almost certainly be dramatically different.
> >We may try to function in the same boat but we may not be able to all
> >row in the same direction. The conceptual framework and tools used in
> >the ICU will be very different from those used in the ambulatory
> >setting. Those used in populations of patients who are sick
> >(symptomatic) will be different from those who are ostensibly well.
> >
> >In the ambulatory setting perhaps one approach to get at frequency
> >would be to look at patient reported outcomes and start with the
> >premise that a portion of bad outcomes will come as a consequence of
> >diagnostic error. With the deployment of CG-cahps questionnaires
> >broadly, perhaps patient reporting can point us to where we should be
> >looking? I do not see much hope for physicians to robustly recognize
> >and report their diagnostic errors.
> >
> >
> >Robert A. Swerlick, MD
> >
> >> On Dec 15, 2013, at 3:10 PM, "Bradford Winters" <bwinters at JHMI.EDU>
> >>wrote:
> >>
> >> In end we may need to measure the surrogates Mark describes for the
> >> 
> >>numerators. Actually counting them would be laborious even if we had
> >>a clear way of flagging them. Plus the cultural resistance would be
> >>nearly insurmountable. Surrogates would help get by that to some
> >>degree but we need clear links between the surrogates and the
> >>diagnostic errors we can measures. Perhaps making those links is the
> >> first step; choose some straight forward ones like mi, pe bleeding
> >>and find surrogates that predict them and move from there. Of course
> >>we never may be sure about the predictive links to other diagnostic
> >>errors
> >>
> >> Sent from my iPhone
> >>
> >>> On Dec 15, 2013, at 10:48 AM, "Graber, Mark" <Mark.Graber at VA.GOV>
> >>>wrote:
> >>>
> >>> The problem of not being able to count diagnostic errors is one
> >>> that 
> >>>keeps me up at night.  Ross is correct in describing the
> >>>fundamental challenges that arise in trying to establish the
> >>>numerator.  If its true that 'you can't improve what you can't
> >>>measure', that will leave diagnostic error at the starting gate.
> >>>
> >>> I've reluctantly come to accept the fact that he is correct - we 
> >>>really CAN'T measure every diagnostic error.  Its not like falls or
> >>> wrong-site surgery, that are more easily tallied.  But .....  I
> >>>believe it may be possible to establish a quantitative approach
> >>>that's not as good as being able to count them all, but may suffice
> >>> for advancing diagnostic science:
> >>>
> >>>
> >>> *   If we use a specific definition and a defined methodological
> >>>approach, we should be able to reproducibly count certain types of
> >>>errors in certain settings.  Example:  delays in diagnosing colon
> >>>cancer.  We will be able to measure the delays, even if we can't
> >>>measure all the missed diagnoses
> >>>
> >>>
> >>> *   If we focus on specific steps of the diagnostic process, we
> >>> should
> >>>be able to reproducibly count lapses in specific steps.  Example: 
> >>>how many patients don't have a differential diagnosis listed on
> >>>their chart;  How many critical alerts aren't addressed in a timely
> >>>manner.
> >>>
> >>> Our efforts to reduce diagnostic error will be seriously stymied
> >>> if 
> >>>we can't come up with at some ways to quantify the current state of
> >>> affairs and see if interventions have any impact, in a
> >>>quantitative sense.
> >>>
> >>> Mark
> >>>
> >>> Mark L Graber, MD FACP
> >>> Senior Fellow, RTI International
> >>> Professor Emeritus, SUNY Stony Brook School of Medicine Founder
> >>> and President, Society to Improve Diagnosis in Medicine
> >>>
> >>>
> >>> ________________________________
> >>> From: Ross Koppel <rkoppel at SAS.UPENN.EDU>
> >>> Reply-To: Society to Improve Diagnosis in Medicine 
> >>><IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>, Ross Koppel 
> >>><rkoppel at SAS.UPENN.EDU>
> >>> Date: Wed, 11 Dec 2013 22:27:48 -0500
> >>> To: <IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>
> >>> Subject: Re: [IMPROVEDX] Error Rates: Diagnosis--neither numerator
> >>> 
> >>>nor denominator is known
> >>>
> >>> The discussion on measurement of error and error rates is
> >>> wonderful and thoughtful. What seems to be missing from much of
> >>> this conversation, however, is that reality that we usually have
> >>> no measures of the numerator -- the number of errors;  and we have
> >>> an often slippery measure of the denominator-- number of
> >>> diagnoses, number of correct diagnoses determined by case review
> >>> and autopsies, number of correct diagnoses determined by a
> >>> brighter doctor, number of correct dx when the patient was
> >>> discharged (sic), number of med orders, number of med orders
> >>> actually administered,  number of opportunities to order, number
> >>> of patients, number of meds administered correctly, etc. Of
> >>> course, the denominator and the numerator obviously differ by the
> >>> measure on which we focus: Some errors are easy to spot, e.g., 
> >>> leaving hemostats in the thoracic cavity is a classic. Med
> >>> prescription errors are very hard to know.  Patients are sick,
> >>> old, have 5 comorbidities and are on 13 other meds.  Bad things
> >>> happen when we do the right thing, good things happen when we do
> >>> the wrong thing, polypharmacy is pandemic, what is ordered may not
> >>> have been administered, no one knows the drug-drug-drug
> >>> interactions among 13 drugs, etc.     Case review is dependent on
> >>> the right Dx (tautology alert) and knowing which tests to order,
> >>> etc.
> >>>
> >>> And then we come to Dx errors. Again, some are easy, but, as  
> >>>demonstrated powerfully by this group's insightful conversation,
> >>>many  are profoundly hard to determine...especially in the first
> >>>set of  iterations.  Some may be impossible. Thus, the  discussion
> >>>of error  rates makes me cry for a massive dose of methodological
> >>>caution and even  more humility.
> >>>
> >>> None of this is to say the discussion is neither productive nor  
> >>>exciting.  Of course not!  But the epistemological issues here are 
> >>> beyond most of the problems of science.  One of the reasons I so
> >>>respect  medical thinking is because physicians must deal with so
> >>>many unknowns  and uncertainties.  My colleague, Renee Fox, noted
> >>>that one of the first  things we teach a young doctor is to
> >>>distinguish between what he/she  does not know and what Medicine
> >>>does not know.  With diagnostic error,  we have both of those
> >>>factors plus the more vexing problem of what is  knowable and what
> >>>is knowable within the constraints of a specific  patient's life
> >>>and setting.
> >>>
> >>> Caveat: I'm not a physician, but I have spent the last 47 years
> >>> focused on research methods, measurement of error, and statistics.
> >>>
> >>> Ross Koppel, Ph.D. FACMI
> >>> Sociology Dept and Sch. of Medicine
> >>> Senior Fellow, LDI, Wharton
> >>> University of Pennsylvania, Phila, PA 19104-6299
> >>> 215 576 8221 C: 215 518 0134
> >>>
> >>> To unsubscribe from the IMPROVEDX:
> >>> mail to:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG
> >>> or click the following link: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
> >>>
> >>> Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
> >>>
> >>> For additional information and subscription commands, visit:
> >>> http://www.lsoft.com/resources/faq.asp#4A
> >>>
> >>> http://LIST.IMPROVEDIAGNOSIS.ORG/ (with your password)
> >>>
> >>> Visit the searchable archives or adjust your subscription at:
> >>> http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?INDEX
> >>>
> >>> Moderator: Lorri Zipperer Lorri at ZPM1.com, Communication co-chair, 
> >>>Society for Improving Diagnosis in Medicine
> >>>
> >>> To unsubscribe from the IMPROVEDX list, click the following 
> >>>link:<br>  <a 
> >>>href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED
> >>>1= IMP ROVEDX&A=1"
> >>>target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG
> >>>.exe? SUBED1=IMPROVEDX&A=1</a>
> >>> </p>
> >>>
> >>> To unsubscribe from the IMPROVEDX:
> >>> mail to:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG
> >>> or click the following link: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
> >>>
> >>> Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
> >>>
> >>> For additional information and subscription commands, visit:
> >>> http://www.lsoft.com/resources/faq.asp#4A
> >>>
> >>> http://LIST.IMPROVEDIAGNOSIS.ORG/ (with your password)
> >>>
> >>> Visit the searchable archives or adjust your subscription at:
> >>> http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?INDEX
> >>>
> >>> Moderator: Lorri Zipperer Lorri at ZPM1.com, Communication co-chair, 
> >>>Society for Improving Diagnosis in Medicine
> >>>
> >>> To unsubscribe from the IMPROVEDX list, click the following 
> >>>link:<br>  <a 
> >>>href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED
> >>>1= IMP ROVEDX&A=1"
> >>>target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG
> >>>.exe? SUBED1=IMPROVEDX&A=1</a>
> >>> </p>
> >>
> >> To unsubscribe from the IMPROVEDX:
> >> mail to:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG
> >> or click the following link: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
> >>
> >> Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
> >>
> >> For additional information and subscription commands, visit:
> >> http://www.lsoft.com/resources/faq.asp#4A
> >>
> >> http://LIST.IMPROVEDIAGNOSIS.ORG/ (with your password)
> >>
> >> Visit the searchable archives or adjust your subscription at:
> >> http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?INDEX
> >>
> >> Moderator: Lorri Zipperer Lorri at ZPM1.com, Communication co-chair, 
> >>Society for Improving Diagnosis in Medicine
> >>
> >> To unsubscribe from the IMPROVEDX list, click the following
> >> link:<br>  
> >><a 
> >>href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1
> >>=I MPR OVEDX&A=1"
> >>target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.
> >>ex e?S UBED1=IMPROVEDX&A=1</a>
> >> </p>
> >
> >________________________________
> >
> >This e-mail message (including any attachments) is for the sole use
> >of the intended recipient(s) and may contain confidential and
> >privileged information. If the reader of this message is not the
> >intended recipient, you are hereby notified that any dissemination,
> >distribution or copying of this message (including any attachments)
> >is strictly prohibited.
> >
> >If you have received this message in error, please contact the sender
> > by reply e-mail message and destroy all copies of the original
> >message (including attachments).
> >
> >To unsubscribe from the IMPROVEDX:
> >mail to:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG
> >or click the following link: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
> >
> >Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
> >
> >For additional information and subscription commands, visit:
> >http://www.lsoft.com/resources/faq.asp#4A
> >
> >http://LIST.IMPROVEDIAGNOSIS.ORG/ (with your password)
> >
> >Visit the searchable archives or adjust your subscription at:
> >http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?INDEX
> >
> >Moderator: Lorri Zipperer Lorri at ZPM1.com, Communication co-chair,
> >Society for Improving Diagnosis in Medicine
> >
> >To unsubscribe from the IMPROVEDX list, click the following link:<br>
> > <a
> >href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=
> >IM PRO VEDX&A=1"
> >target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.e
> >xe ?SU BED1=IMPROVEDX&A=1</a> </p>
> 
> 
> 
> 
> 
> 
> Moderator: Lorri Zipperer Lorri at ZPM1.com, Communication co-chair,
> Society for Improving Diagnosis in Medicine
> 
> To unsubscribe from the IMPROVEDX list, click the following link:<br>
> <a
> href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=I
> MPROVEDX&A=1"
> target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.ex
> e?SUBED1=IMPROVEDX&A=1</a> </p>
> 
>  -------------------------------------------------------------------
> The information transmitted by this e-mail and any included
> attachments are from ARUP Laboratories and are intended only for the
> recipient. The information contained in this message is confidential
> and may constitute inside or non-public information under
> international, federal, or state securities laws, or protected health
> information and is intended only for the use of the recipient.
> Unauthorized forwarding, printing, copying, distributing, or use of
> such information is strictly prohibited and may be unlawful. If you
> are not the intended recipient, please promptly delete this e-mail and
> notify the sender of the delivery error or you may call ARUP
> Laboratories Compliance Hot Line in Salt Lake City, Utah USA at (+1
> (800) 522-2787 ext. 2100
> 
> 
> 
> 
> 
> 
> Moderator: Lorri Zipperer Lorri at ZPM1.com, Communication co-chair,
> Society for Improving Diagnosis in Medicine
> 
> To unsubscribe from the IMPROVEDX list, click the following link:<br>
> <a
> href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=I
> MPROVEDX&A=1"
> target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.ex
> e?SUBED1=IMPROVEDX&A=1</a> </p>
> 

Robert L Wears, MD, MS, PhD
University of Florida  	Imperial College London
wears at ufl.edu		r.wears at imperial.ac.uk
1-904-244-4405 (ass't)  	+44 (0)791 015 2219
They said it would get worse before it gets better,
and they were right.  







To unsubscribe from the IMPROVEDX list, click the following link:<br>
<a href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1" target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1</a>
</p>



More information about the Test mailing list