Error Rates: Diagnosis--neither numerator nor denominator is known

John Brush jebrush at ME.COM
Mon Dec 16 18:38:27 UTC 2013


I agree with Dr. Papa. Our best way to improve diagnosis is to teach and re-teach the principles of making a diagnosis. This is the "structure", which could be measured. We might be able to construct some "process" measures (e.g. Inclusion of a short differential diagnosis in each diagnostic workup), but measuring "outcomes" is fraught with problems and may cause unintended consequences such as over-diagnosis, excessive testing, and hasty generalization. For outcomes, it would be impossible to measure the denominator or to risk adjust.
Doctors don't need more motivation to achieve optimal diagnostic outcomes. Every doctor knows that missed diagnosis is the most common reason for a malpractice suit.
We should focus on teaching trainees to have an organized and logical approach, grounded in science and probability principles, and grounded in cognitive psychology theory.
John Brush

Sent from my iPad

> On Dec 16, 2013, at 12:31 PM, "Papa, Frank" <Frank.Papa at UNTHSC.EDU> wrote:
> 
> I continue to deeply appreciate the thoughts all contributors are making towards formulating better approaches to measuring the incidence of diagnostic error in the patient care environment - perhaps the most complex of all quality assurance/assessment issues in medicine. However, I'd like to suggest that some attention also be directed at approaches used to teach to DDX competence, and measure DDX performance (accuracy vs error), in the medical training environment.
> 
> It remains a very curious thing that to date, medical education has never formulated and broadly embraced, a codified approach to DDX training and assessment. This is not to say that we lack cognitively-grounded theories and mathematically-anchored approaches that could inform DDX instruction and assessment. 
> 
> Rather, and to just begin such a discussion, I want to suggest that in the pre-clinical training environment, few if any medical schools seem to have coursework that is not only focused on DDX training and assessment, but also leads to a grade reflecting DDX capabilities. Further, Level/Step 1 boards continue to produce reports in terms of levels of achievement in basic science disciplines and broadly defined, clinical science arenas. What does performance in these various disciplines and arenas tell us about a candidate's evolving DDX capabilities/competencies?
> 
> I take as a given that the most critical of all skills to train to and adequately assess during medical training is DDX capabilities/competencies. SIDM can and should direct a portion of its efforts at providing leadership in how undergraduate medical training programs might develop and/or work towards a codified approach to DDX training and assessment, and, pushing licensing boards to construct formal metrics reflecting DDX performance capabilities as early as in its Level/Step 1 examinations.
> 
> 
> Frank J Papa, DO, PhD
> Professor, Medical Education and Emergency Medicine
> Director, TCOM Academy of Medical Educators
> Associate Dean, Curricular Design and Faculty Development
> University of North Texas Health Science Center
> 
> 
> 
> -----Original Message-----
> From: Bob Latino [mailto:blatino at RELIABILITY.COM] 
> Sent: Monday, December 16, 2013 10:31 AM
> To: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
> Subject: Re: [IMPROVEDX] Error Rates: Diagnosis--neither numerator nor denominator is known
> 
> As a newcomer to your group, I am learning a lot as a 'lurker' and listening to the challenges and reasoning associated with diagnosis related issues. Thank you.
> 
> I will again preface that I am not a clinician or an epidemiologist. I am an investigator who analyzes individual cases across a spectrum of industries, involving an undesirable outcome of some kind.  Sometimes such outcomes are related to diagnosis issues and sometimes they are not.  That was the basis of my initial question to this group about any studies that would discern the frequency of undesirable outcomes where diagnosis issues were a contributing factor.  I am learning that such a statistic may not be available and if it were, it may not be credible because of the considerable variability of circumstances involved.
> 
> I also know myself it may not be credible because of a lack of identifying diagnosis error as a factor in an investigation.  Just because it was not identified in a Root Cause Analysis (RCA), does not mean it was not a factor.  There are many reasons that such an issue may not have been included in the analysis, but that is another thread.
> 
> When I collaborate with colleagues in the research field, I liken my work to being a single data point in an overall trend.  This represents single cases that are looked at great in-depth.
> 
> Given this narrow perspective, I spend quite a deal of time trying to understand individual decision-making no matter the occupation.  When focusing on the human being as the common denominator (when inappropriate decisions are made), why did the decision-maker think it was the right decision at the time? 
> 
> I have been listening to your very interesting debates.  You all are obviously clinical professionals who have likely been in such circumstances and can very much relate to a decision-maker who made an inappropriate decision at the time. When looking at this moment in time and putting ourselves in the position of the decision-maker (and trying to avoid hindsight bias), are there commonalities amongst these decision-makers that could be categorized during an analysis/investigation and then used for trending purposes?
> 
> Did the decision-maker have the adequate credentials and qualifications to make the correct decision?
> Did the decision-maker have the adequate information in which to make the correct decision?
> Was the decision-maker under stress (e.g. time pressure) or duress (e.g. - fear of repercussion) which could impair their decision-making process?
> Was the decision-maker over-confident in their ability to make the correct decision?
> Did the decision-maker have low alertness due to illness, fatigue or boredom?
> 
> This is just a sampling of the questions that go through my mind based on past investigations across many industries.  
> 
> I have attached a sample job aid we use to assist our clients in considering why people may have made an inappropriate decision at the time.  By far this is not an exhaustive list but just a job aid to help people consider things they may not normally consider.
> 
> Does the construct of such a tool for specific application in understanding diagnosis errors, have any merit?  Given the complexity of the variables involved in diagnosis error, please understand this is just a generic sample for looking at decision-making in general.  How would the causes of errors in diagnosis differ from the considerations on this generic job aid?
> 
> In my world I see diagnosis error as potential contributing factor which requires a deeper drill down to understand the reasoning of the errors themselves. The end result will be some type of undesirable outcome.  Usually the outcome is a near miss (if we are lucky) or a degree of patient harm.
> 
> Thanks for your patience while I learn how to fit into this bigger picture of diagnosis error.  
> 
> Robert (Bob) J. Latino
> CEO
> Reliability Center, Inc., P.O. Box 1421, Hopewell, VA  23860
> (O) 804.458.0645  (F) 804.452.2119
> blatino at reliability.com l http://www.reliability.com 
> 
> 
> -----Original Message-----
> From: Swerlick, Robert A [mailto:rswerli at EMORY.EDU]
> Sent: Sunday, December 15, 2013 11:32 PM
> To: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
> Subject: Re: [IMPROVEDX] Error Rates: Diagnosis--neither numerator nor denominator is known
> 
> While the issue of diagnostic error is the central focus of this organization, we continue to stumble when it comes to understanding frequency and definitions. We all know it exists and can identify specific examples within our own worlds but are hard pressed to come up with great approaches to measure.
> 
> One of the strengths of the SIDM is the diversity of participants. It is also one of the weaknesses because the nature is diagnostic errors observed and/or experienced varies hugely across the spectrum of SIDM members. The is the inpatient/outpatient divide, the acute care/chronic care divide, urgent/non-urgent divide, and practitioner/patient divide. I also think there is a divide regarding under-diagnosis (missed diagnosis) vs. over-diagnosis often playing out in populations of patients who might be viewed as well (no symptoms).
> 
> The priorities of these different constituencies may be very different and we may be best served by recognizing up front that the challenges and priorities and tools used to get at diagnostic errors in these scenarios will almost certainly be dramatically different. We may try to function in the same boat but we may not be able to all row in the same direction. The conceptual framework and tools used in the ICU will be very different from those used in the ambulatory setting. Those used in populations of patients who are sick (symptomatic) will be different from those who are ostensibly well.
> 
> In the ambulatory setting perhaps one approach to get at frequency would be to look at patient reported outcomes and start with the premise that a portion of bad outcomes will come as a consequence of diagnostic error. With the deployment of CG-cahps questionnaires broadly, perhaps patient reporting can point us to where we should be looking? I do not see much hope for physicians to robustly recognize and report their diagnostic errors.
> 
> 
> Robert A. Swerlick, MD
> 
>> On Dec 15, 2013, at 3:10 PM, "Bradford Winters" <bwinters at JHMI.EDU> wrote:
>> 
>> In end we may need to measure the surrogates Mark describes for the 
>> numerators. Actually counting them would be laborious even if we had a 
>> clear way of flagging them. Plus the cultural resistance would be 
>> nearly insurmountable. Surrogates would help get by that to some 
>> degree but we need clear links between the surrogates and the 
>> diagnostic errors we can measures. Perhaps making those links is the 
>> first step; choose some straight forward ones like mi, pe bleeding and 
>> find surrogates that predict them and move from there. Of course we 
>> never may be sure about the predictive links to other diagnostic 
>> errors
>> 
>> Sent from my iPhone
>> 
>>> On Dec 15, 2013, at 10:48 AM, "Graber, Mark" <Mark.Graber at VA.GOV> wrote:
>>> 
>>> The problem of not being able to count diagnostic errors is one that keeps me up at night.  Ross is correct in describing the fundamental challenges that arise in trying to establish the numerator.  If its true that 'you can't improve what you can't measure', that will leave diagnostic error at the starting gate.
>>> 
>>> I've reluctantly come to accept the fact that he is correct - we really CAN'T measure every diagnostic error.  Its not like falls or wrong-site surgery, that are more easily tallied.  But .....  I believe it may be possible to establish a quantitative approach that's not as good as being able to count them all, but may suffice for advancing diagnostic science:
>>> 
>>> 
>>> *   If we use a specific definition and a defined methodological approach, we should be able to reproducibly count certain types of errors in certain settings.  Example:  delays in diagnosing colon cancer.  We will be able to measure the delays, even if we can't measure all the missed diagnoses
>>> 
>>> 
>>> *   If we focus on specific steps of the diagnostic process, we should be able to reproducibly count lapses in specific steps.  Example:  how many patients don't have a differential diagnosis listed on their chart;  How many critical alerts aren't addressed in a timely manner.
>>> 
>>> Our efforts to reduce diagnostic error will be seriously stymied if we can't come up with at some ways to quantify the current state of affairs and see if interventions have any impact, in a quantitative sense.
>>> 
>>> Mark
>>> 
>>> Mark L Graber, MD FACP
>>> Senior Fellow, RTI International
>>> Professor Emeritus, SUNY Stony Brook School of Medicine Founder and 
>>> President, Society to Improve Diagnosis in Medicine
>>> 
>>> 
>>> ________________________________
>>> From: Ross Koppel <rkoppel at SAS.UPENN.EDU>
>>> Reply-To: Society to Improve Diagnosis in Medicine 
>>> <IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>, Ross Koppel 
>>> <rkoppel at SAS.UPENN.EDU>
>>> Date: Wed, 11 Dec 2013 22:27:48 -0500
>>> To: <IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>
>>> Subject: Re: [IMPROVEDX] Error Rates: Diagnosis--neither numerator 
>>> nor denominator is known
>>> 
>>> The discussion on measurement of error and error rates is wonderful 
>>> and thoughtful.
>>> What seems to be missing from much of this conversation, however, is 
>>> that reality that we usually have no measures of the numerator -- the 
>>> number of errors;  and we have an often slippery measure of the
>>> denominator-- number of diagnoses, number of correct diagnoses 
>>> determined by case review and autopsies, number of correct diagnoses 
>>> determined by a brighter doctor, number of correct dx when the 
>>> patient was discharged (sic), number of med orders, number of med 
>>> orders actually administered,  number of opportunities to order, 
>>> number of patients, number of meds administered correctly, etc.
>>> Of course, the denominator and the numerator obviously differ by the 
>>> measure on which we focus:
>>> Some errors are easy to spot, e.g.,  leaving hemostats in the 
>>> thoracic cavity is a classic.
>>> Med prescription errors are very hard to know.  Patients are sick, 
>>> old, have 5 comorbidities and are on 13 other meds.  Bad things 
>>> happen when we do the right thing, good things happen when we do the 
>>> wrong thing, polypharmacy is pandemic, what is ordered may not have 
>>> been administered, no one knows the drug-drug-drug interactions among 13
>>> drugs, etc.     Case review is dependent on the right Dx (tautology
>>> alert) and knowing which tests to order, etc.
>>> 
>>> And then we come to Dx errors. Again, some are easy, but, as 
>>> demonstrated powerfully by this group's insightful conversation, many 
>>> are profoundly hard to determine...especially in the first set of 
>>> iterations.  Some may be impossible. Thus, the  discussion of error 
>>> rates makes me cry for a massive dose of methodological caution and 
>>> even more humility.
>>> 
>>> None of this is to say the discussion is neither productive nor 
>>> exciting.  Of course not!  But the epistemological issues here are 
>>> beyond most of the problems of science.  One of the reasons I so 
>>> respect medical thinking is because physicians must deal with so many 
>>> unknowns and uncertainties.  My colleague, Renee Fox, noted that one 
>>> of the first things we teach a young doctor is to distinguish between 
>>> what he/she does not know and what Medicine does not know.  With 
>>> diagnostic error, we have both of those factors plus the more vexing 
>>> problem of what is knowable and what is knowable within the 
>>> constraints of a specific patient's life and setting.
>>> 
>>> Caveat: I'm not a physician, but I have spent the last 47 years 
>>> focused on research methods, measurement of error, and statistics.
>>> 
>>> Ross Koppel, Ph.D. FACMI
>>> Sociology Dept and Sch. of Medicine
>>> Senior Fellow, LDI, Wharton
>>> University of Pennsylvania, Phila, PA 19104-6299
>>> 215 576 8221 C: 215 518 0134
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Moderator: Lorri Zipperer Lorri at ZPM1.com, Communication co-chair, 
>>> Society for Improving Diagnosis in Medicine
>>> 
>>> To unsubscribe from the IMPROVEDX list, click the following link:<br> 
>>> <a 
>>> href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=
>>> IMPROVEDX&A=1" 
>>> target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.e
>>> xe?SUBED1=IMPROVEDX&A=1</a>
>>> </p>
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Moderator: Lorri Zipperer Lorri at ZPM1.com, Communication co-chair, 
>>> Society for Improving Diagnosis in Medicine
>>> 
>>> To unsubscribe from the IMPROVEDX list, click the following link:<br> 
>>> <a 
>>> href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=
>>> IMPROVEDX&A=1" 
>>> target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.e
>>> xe?SUBED1=IMPROVEDX&A=1</a>
>>> </p>
>> 
>> 
>> 
>> 
>> 
>> 
>> Moderator: Lorri Zipperer Lorri at ZPM1.com, Communication co-chair, 
>> Society for Improving Diagnosis in Medicine
>> 
>> To unsubscribe from the IMPROVEDX list, click the following link:<br> 
>> <a 
>> href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=I
>> MPROVEDX&A=1" 
>> target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.ex
>> e?SUBED1=IMPROVEDX&A=1</a>
>> </p>
> 
> ________________________________
> 
> This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited.
> 
> If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments).
> 
> 
> 
> 
> 
> 
> Moderator: Lorri Zipperer Lorri at ZPM1.com, Communication co-chair, Society for Improving Diagnosis in Medicine
> 
> To unsubscribe from the IMPROVEDX list, click the following link:<br> <a href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1" target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1</a>
> </p>
> 
> 
> 
> 
> 
> 
> Moderator: Lorri Zipperer Lorri at ZPM1.com, Communication co-chair, Society for Improving Diagnosis in Medicine
> 
> To unsubscribe from the IMPROVEDX list, click the following link:<br> <a href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1" target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1</a>
> </p>
> 
> 
> 
> 
> 
> 
> Moderator: Lorri Zipperer Lorri at ZPM1.com, Communication co-chair, Society for Improving Diagnosis in Medicine
> 
> To unsubscribe from the IMPROVEDX list, click the following link:<br>
> <a href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1" target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1</a>
> </p>










More information about the Test mailing list