Error Rates: Diagnosis--neither numerator nor denominator is known

Giuliano, Michael M.D. MGiuliano at HACKENSACKUMC.ORG
Mon Dec 16 21:08:46 UTC 2013


  I have a concrete example from the Neonatal ICU. We developed a 17 item checklist of items that we thought would be red flags for possible dx error. We used the check list at each morning and evening sign out. The list included patients with a new significant diagnosis or specific issues like a Pneumothorax.  After a trigger was noted the signing out MD was briefly interviewed to determine if they thought the dx could have been made earlier. The charts were then summarized in a standard way and reviewed by two on site reviewers. Any case that was possibly a dx error was then sent to a blinded 5 person panel from outside the institution. Any case that received 3 or more votes as an error was counted.  We used patient days as our denominator since infants stayed in the NICU for many days and could be subject to dx error at any point in their stay. The error rate was calculated as errors per patient days.
 Each setting would need to think about how this might work but it is one way to try to measure the error rate.
Michael Giuliano
Director Neonatology
Hackensack University Medical center, NJ
Rutgers, NJMS


-----Original Message-----
From: Bradford Winters [mailto:bwinters at JHMI.EDU]
Sent: Sunday, December 15, 2013 7:46 PM
To: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
Subject: Re: [IMPROVEDX] Error Rates: Diagnosis--neither numerator nor denominator is known

I was thinking that if we could develop models for predictors for some of the major misdiagnoses we could follow the incidence of these predictors showing up. We could consider them at least near events if they carry a high risk of progressing to the misdiagnosis. I need to think about it a bit more but if we could codify how we miss PE or MI in the majority of cases we could use those as surrogates.

Sent from my iPhone

> On Dec 15, 2013, at 7:35 PM, "Graber, Mark" <Mark.Graber at VA.GOV> wrote:
>
> Brad - can you give us a concrete example of a surrogate and the kind of linkages you're thinking about ?  Seems like a promising approach.
>
>
> ________________________________
> From: Bradford Winters <bwinters at jhmi.edu>
> Date: Sun, 15 Dec 2013 15:08:44 -0500
> To: Society to Improve Diagnosis in Medicine <IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>, "mark.graber at va.gov" <mark.graber at va.gov>
> Cc: <IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>
> Subject: Re: [IMPROVEDX] Error Rates: Diagnosis--neither numerator nor denominator is known
>
> In end we may need to measure the surrogates Mark describes for the numerators. Actually counting them would be laborious even if we had a clear way of flagging them. Plus the cultural resistance would be nearly insurmountable. Surrogates would help get by that to some degree but we need clear links between the surrogates and the diagnostic errors we can measures. Perhaps making those links is the first step; choose some straight forward ones like mi, pe bleeding and find surrogates that predict them and move from there. Of course we never may be sure about the predictive links to other diagnostic errors
>
> Sent from my iPhone
>
>> On Dec 15, 2013, at 10:48 AM, "Graber, Mark" <Mark.Graber at VA.GOV> wrote:
>>
>> The problem of not being able to count diagnostic errors is one that keeps me up at night.  Ross is correct in describing the fundamental challenges that arise in trying to establish the numerator.  If its true that 'you can't improve what you can't measure', that will leave diagnostic error at the starting gate.
>>
>> I've reluctantly come to accept the fact that he is correct - we really CAN'T measure every diagnostic error.  Its not like falls or wrong-site surgery, that are more easily tallied.  But .....  I believe it may be possible to establish a quantitative approach that's not as good as being able to count them all, but may suffice for advancing diagnostic science:
>>
>>
>> *   If we use a specific definition and a defined methodological approach, we should be able to reproducibly count certain types of errors in certain settings.  Example:  delays in diagnosing colon cancer.  We will be able to measure the delays, even if we can't measure all the missed diagnoses
>>
>>
>> *   If we focus on specific steps of the diagnostic process, we should be able to reproducibly count lapses in specific steps.  Example:  how many patients don't have a differential diagnosis listed on their chart;  How many critical alerts aren't addressed in a timely manner.
>>
>> Our efforts to reduce diagnostic error will be seriously stymied if we can't come up with at some ways to quantify the current state of affairs and see if interventions have any impact, in a quantitative sense.
>>
>> Mark
>>
>> Mark L Graber, MD FACP
>> Senior Fellow, RTI International
>> Professor Emeritus, SUNY Stony Brook School of Medicine
>> Founder and President, Society to Improve Diagnosis in Medicine
>>
>>
>> ________________________________
>> From: Ross Koppel <rkoppel at SAS.UPENN.EDU>
>> Reply-To: Society to Improve Diagnosis in Medicine <IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>, Ross Koppel <rkoppel at SAS.UPENN.EDU>
>> Date: Wed, 11 Dec 2013 22:27:48 -0500
>> To: <IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>
>> Subject: Re: [IMPROVEDX] Error Rates: Diagnosis--neither numerator nor denominator is known
>>
>> The discussion on measurement of error and error rates is wonderful and
>> thoughtful.
>> What seems to be missing from much of this conversation, however, is
>> that reality that we usually have no measures of the numerator -- the
>> number of errors;  and we have an often slippery measure of the
>> denominator-- number of diagnoses, number of correct diagnoses
>> determined by case review and autopsies, number of correct diagnoses
>> determined by a brighter doctor, number of correct dx when the patient
>> was discharged (sic), number of med orders, number of med orders
>> actually administered,  number of opportunities to order, number of
>> patients, number of meds administered correctly, etc.
>> Of course, the denominator and the numerator obviously differ by the
>> measure on which we focus:
>> Some errors are easy to spot, e.g.,  leaving hemostats in the thoracic
>> cavity is a classic.
>> Med prescription errors are very hard to know.  Patients are sick, old,
>> have 5 comorbidities and are on 13 other meds.  Bad things happen when
>> we do the right thing, good things happen when we do the wrong thing,
>> polypharmacy is pandemic, what is ordered may not have been
>> administered, no one knows the drug-drug-drug interactions among 13
>> drugs, etc.     Case review is dependent on the right Dx (tautology
>> alert) and knowing which tests to order, etc.
>>
>> And then we come to Dx errors. Again, some are easy, but, as
>> demonstrated powerfully by this group's insightful conversation, many
>> are profoundly hard to determine...especially in the first set of
>> iterations.  Some may be impossible. Thus, the  discussion of error
>> rates makes me cry for a massive dose of methodological caution and even
>> more humility.
>>
>> None of this is to say the discussion is neither productive nor
>> exciting.  Of course not!  But the epistemological issues here are
>> beyond most of the problems of science.  One of the reasons I so respect
>> medical thinking is because physicians must deal with so many unknowns
>> and uncertainties.  My colleague, Renee Fox, noted that one of the first
>> things we teach a young doctor is to distinguish between what he/she
>> does not know and what Medicine does not know.  With diagnostic error,
>> we have both of those factors plus the more vexing problem of what is
>> knowable and what is knowable within the constraints of a specific
>> patient's life and setting.
>>
>> Caveat: I'm not a physician, but I have spent the last 47 years focused
>> on research methods, measurement of error, and statistics.
>>
>> Ross Koppel, Ph.D. FACMI
>> Sociology Dept and Sch. of Medicine
>> Senior Fellow, LDI, Wharton
>> University of Pennsylvania, Phila, PA 19104-6299
>> 215 576 8221 C: 215 518 0134
>>
>> To unsubscribe from the IMPROVEDX:
>> mail to:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG
>> or click the following link: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
>>
>> Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
>>
>> For additional information and subscription commands, visit:
>> http://www.lsoft.com/resources/faq.asp#4A
>>
>> http://LIST.IMPROVEDIAGNOSIS.ORG/ (with your password)
>>
>> Visit the searchable archives or adjust your subscription at:
>> http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?INDEX
>>
>> Moderator: Lorri Zipperer Lorri at ZPM1.com, Communication co-chair, Society for Improving Diagnosis in Medicine
>>
>> To unsubscribe from the IMPROVEDX list, click the following link:<br>
>> <a href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1" target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1</a>
>> </p>
>>
>> To unsubscribe from the IMPROVEDX:
>> mail to:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG
>> or click the following link: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
>>
>> Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
>>
>> For additional information and subscription commands, visit:
>> http://www.lsoft.com/resources/faq.asp#4A
>>
>> http://LIST.IMPROVEDIAGNOSIS.ORG/ (with your password)
>>
>> Visit the searchable archives or adjust your subscription at:
>> http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?INDEX
>>
>> Moderator: Lorri Zipperer Lorri at ZPM1.com, Communication co-chair, Society for Improving Diagnosis in Medicine
>>
>> To unsubscribe from the IMPROVEDX list, click the following link:<br>
>> <a href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1" target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1</a>
>> </p>
>
> To unsubscribe from the IMPROVEDX:
> mail to:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG
> or click the following link: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
>
> Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG
>
> For additional information and subscription commands, visit:
> http://www.lsoft.com/resources/faq.asp#4A
>
> http://LIST.IMPROVEDIAGNOSIS.ORG/ (with your password)
>
> Visit the searchable archives or adjust your subscription at:
> http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?INDEX
>
> Moderator: Lorri Zipperer Lorri at ZPM1.com, Communication co-chair, Society for Improving Diagnosis in Medicine
>
> To unsubscribe from the IMPROVEDX list, click the following link:<br>
> <a href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1" target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1</a>
> </p>

To unsubscribe from the IMPROVEDX:
mail to:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG
or click the following link: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG

Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG

For additional information and subscription commands, visit:
http://www.lsoft.com/resources/faq.asp#4A

http://LIST.IMPROVEDIAGNOSIS.ORG/ (with your password)

Visit the searchable archives or adjust your subscription at:
http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?INDEX

Moderator: Lorri Zipperer Lorri at ZPM1.com, Communication co-chair, Society for Improving Diagnosis in Medicine

To unsubscribe from the IMPROVEDX list, click the following link:<br>
<a href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1" target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1</a>
</p>
************************************************************************************************************* Confidentiality Notice: This email message and any attachments from HackensackUMC are confidential and for the sole use of the intended recipient. This communication may contain Protected Health Information ("PHI"). PHI is confidential information that may only be used or disclosed in accordance with applicable law. There are penalties under the law for the improper use or further disclosure of PHI. If you are not the intended recipient of this email or the employee or agent responsible for delivering the communication to the intended recipient, then you may not read, copy, distribute or otherwise use or disclose the information contained in this message. If you received this message in error, please notify us by telephone at 551-996-2000 or by email to Postmaster at HackensackUMC.org Please indicate that you were not the intended recipient, and confirm that you have deleted the original message. Please do not retransmit the contents of the message. Thank you. ************************************************************************************************************* About Hackensack University Medical Center HackensackUMC, a nonprofit teaching and research hospital located in Bergen County, NJ, is the largest provider of inpatient and outpatient services in the state. It is the flagship hospital of the Hackensack University Health Network, one of the largest health networks in the state comprised of 1,140 beds, nearly 10,000 employees and 2,200 credentialed physicians. HackensackUMC was listed as the number one hospital in New Jersey and one of the top three New York metro area hospitals in the U.S. News & World Report’s Best Hospital rankings for 2013-14. It also received 10 national rankings in: Cancer; Cardiology & Heart Surgery; Ear, Nose & Throat; Gastroenterology & GI Surgery; Geriatrics; Neurology & Neurosurgery; Orthopedics; Pulmonology; Urology; and the Joseph M. Sanzari Children’s Hospital ranked as one of the Top 50 Best Children’s Hospitals for Neurology and Neurosurgery in the 2013-14 Best Children’s Hospitals list. HackensackUMC is among Healthgrades® America’s Best 100 Hospitals in 10 different areas – more than any other hospital in the nation, is one of Healthgrades America's 50 Best Hospitals™ for seven years in a row, and received the Healthgrades Distinguished Hospital Award for Clinical Excellence™ 11 years in a row. The medical center has also been named a Leapfrog Top Hospital, and received 19 Gold Seals of Approval™ by the Joint Commission – more than any other hospital in the country. It was the first hospital in New Jersey and second in the nation to become a Magnet® recognized hospital for nursing excellence. HackensackUMC is the Hometown Hospital of the New York Giants and the New York Red Bulls, and remains committed to its community through fundraising and community events. To learn more, visit: www.HackensackUMC.org.

To unsubscribe from the IMPROVEDX:
mail to:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG
or click the following link: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG

Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG

For additional information and subscription commands, visit:
http://www.lsoft.com/resources/faq.asp#4A

http://LIST.IMPROVEDIAGNOSIS.ORG/ (with your password)

Visit the searchable archives or adjust your subscription at:
http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?INDEX

Moderator: Lorri Zipperer Lorri at ZPM1.com, Communication co-chair, Society for Improving Diagnosis in Medicine

To unsubscribe from the IMPROVEDX list, click the following link:<br>
<a href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1" target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1</a>
</p>


More information about the Test mailing list