Error Rates: Diagnosis--neither numerator nor denominator is known

Swerlick, Robert A rswerli at EMORY.EDU
Mon Dec 16 16:10:42 UTC 2013


I ill have to respectfully disagree that there can be any consensus in this arena of cancer screening and diagnostic error. If we are to use a surrogate measure such as measuring how aggressively we screen a given population, increasing success as measured by such a surrogate may easily increase the rate of diagnostic error in the form of over-diagnosis.

Dr. Strull notes that there is excellent data linking overall prognosis to early stage of diagnosis but this may be related more to the diagnoses of benign disease as malignant as opposed to saving any lives. More early cancer diagnosis may simply reflect a more aggressive approach to screenign and may yield little of not value to patients. This is a morass. I suspect the SIDM may be equally divided in terms of those who might believe that cancer screening is a source vs. a solution to the diagnostic error problem.

Robert A. Swerlick, MD

On Dec 16, 2013, at 10:05 AM, "Hayward, Rodney (Rod)" <rhayward at MED.UMICH.EDU<mailto:rhayward at MED.UMICH.EDU>> wrote:

With regards to the below proposal, the ratio of late to early cancers is known to be highly vulnerable to detection bias. The epidemiologically correct rate to examine is the absolute hazard rate of advanced cancers, but unfortunately, that will vary with the patient population’s underlying risk for that cancer, making it problematic for cross system comparisons (though it can be used to examine times series data examining within system improvements). This problem has vexed epidemiologists for a long time, and I see no solution that would enable the worthy goal of trying to reliably profile health system’s relative success in early cancer detection, other than the current, and admittedly suboptimal, approach of examining the rate of using screening methods that have good evidence for improving outcomes. This paradox -- that outcomes are what we care about but processes are the best feasible measure of comparative quality (when carefully selected based on RCT evidence that examined outcomes) -- is true for most quality profiling problems.

Rod
_____________________________________________________________
Rod Hayward, MD
Director, Robert Wood Johnson Foundation Clinical Scholars®
Professor of Medicine & Public Health
University of Michigan
Assistant: Brittany Weatherwax (bweather at umich.edu<mailto:bweather at umich.edu>, Ph# 734/647-4844)

Senior Investigator
VA Center for Clinical Management Research
VA Ann Arbor Healthcare System

<image001.png>

Clinical Scholars® is a national program of the Robert Wood Johnson Foundation supported in part through
a collaboration with the U.S. Department of Veterans Affairs (http://rwjcsp.unc.edu<http://rwjcsp.unc.edu/>).



From: William.Strull at KP.ORG<mailto:William.Strull at KP.ORG> [mailto:William.Strull at KP.ORG]
Sent: Monday, December 16, 2013 1:01 AM
To: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>
Subject: Re: [IMPROVEDX] Error Rates: Diagnosis--neither numerator nor denominator is known

I agree with the importance of examining how diagnostic processes impact Donabedian's structure, process, and outcomes, but think that the outcomes data speak most clearly to our operational colleagues and to our patients.  To that extent, in the realm of cancer, there is excellent data linking stage at diagnosis to prognosis.  So one metric about reliable diagnosis across health care systems could reflect the proportion of early stage (stage I and II, generally) versus late stage (stages III and IV) diagnosis of the screenable cancers (breast, colon, prostate, cervical, and potentially lung), as well as cancer-specific mortality rates.  This could be highly revealing in regard to how our systems are supporting the health of our patients.  Similarly, outcomes metrics related to cardiovascular and cerebrovascular disease....


William Strull MD
Medical Director, Quality and Patient Safety
Kaiser Permanente

The Permanente Federation, LLC
One Kaiser Plaza, 23B
Oakland, California 94612
510-271-5987 (office)
8-423-5987 (tie-line)
510-271-6642 (fax)
415-601-6013 (mobile phone)

Debra C. Costa (assistant)
debra.c.costa at kp.org<mailto:debra.c.costa at kp.org>
510-271-6031

NOTICE TO RECIPIENT:  If you are not the intended recipient of this e-mail, you are prohibited from sharing, copying, or otherwise using or disclosing its contents.  If you have received this e-mail in error, please notify the sender immediately by reply e-mail and permanently delete this e-mail and any attachments without reading, forwarding or saving them.  Thank you.



From:        "Graber, Mark" <Mark.Graber at VA.GOV<mailto:Mark.Graber at VA.GOV>>
To:        IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>
Date:        12/15/2013 08:01 AM
Subject:        Re: [IMPROVEDX] Error Rates: Diagnosis--neither numerator nor denominator is known

________________________________



The problem of not being able to count diagnostic errors is one that keeps me up at night.  Ross is correct in describing the fundamental challenges that arise in trying to establish the numerator.  If its true that 'you can't improve what you can't measure', that will leave diagnostic error at the starting gate.

I've reluctantly come to accept the fact that he is correct - we really CAN'T measure every diagnostic error.  Its not like falls or wrong-site surgery, that are more easily tallied.  But .....  I believe it may be possible to establish a quantitative approach that's not as good as being able to count them all, but may suffice for advancing diagnostic science:


*   If we use a specific definition and a defined methodological approach, we should be able to reproducibly count certain types of errors in certain settings.  Example:  delays in diagnosing colon cancer.  We will be able to measure the delays, even if we can't measure all the missed diagnoses


*   If we focus on specific steps of the diagnostic process, we should be able to reproducibly count lapses in specific steps.  Example:  how many patients don't have a differential diagnosis listed on their chart;  How many critical alerts aren't addressed in a timely manner.

Our efforts to reduce diagnostic error will be seriously stymied if we can't come up with at some ways to quantify the current state of affairs and see if interventions have any impact, in a quantitative sense.

Mark

Mark L Graber, MD FACP
Senior Fellow, RTI International
Professor Emeritus, SUNY Stony Brook School of Medicine
Founder and President, Society to Improve Diagnosis in Medicine


________________________________
From: Ross Koppel <rkoppel at SAS.UPENN.EDU<mailto:rkoppel at SAS.UPENN.EDU>>
Reply-To: Society to Improve Diagnosis in Medicine <IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>>, Ross Koppel <rkoppel at SAS.UPENN.EDU<mailto:rkoppel at SAS.UPENN.EDU>>
Date: Wed, 11 Dec 2013 22:27:48 -0500
To: <IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>>
Subject: Re: [IMPROVEDX] Error Rates: Diagnosis--neither numerator nor denominator is known

The discussion on measurement of error and error rates is wonderful and
thoughtful.
What seems to be missing from much of this conversation, however, is
that reality that we usually have no measures of the numerator -- the
number of errors;  and we have an often slippery measure of the
denominator-- number of diagnoses, number of correct diagnoses
determined by case review and autopsies, number of correct diagnoses
determined by a brighter doctor, number of correct dx when the patient
was discharged (sic), number of med orders, number of med orders
actually administered,  number of opportunities to order, number of
patients, number of meds administered correctly, etc.
Of course, the denominator and the numerator obviously differ by the
measure on which we focus:
Some errors are easy to spot, e.g.,  leaving hemostats in the thoracic
cavity is a classic.
Med prescription errors are very hard to know.  Patients are sick, old,
have 5 comorbidities and are on 13 other meds.  Bad things happen when
we do the right thing, good things happen when we do the wrong thing,
polypharmacy is pandemic, what is ordered may not have been
administered, no one knows the drug-drug-drug interactions among 13
drugs, etc.     Case review is dependent on the right Dx (tautology
alert) and knowing which tests to order, etc.

And then we come to Dx errors. Again, some are easy, but, as
demonstrated powerfully by this group's insightful conversation, many
are profoundly hard to determine...especially in the first set of
iterations.  Some may be impossible. Thus, the  discussion of error
rates makes me cry for a massive dose of methodological caution and even
more humility.

None of this is to say the discussion is neither productive nor
exciting.  Of course not!  But the epistemological issues here are
beyond most of the problems of science.  One of the reasons I so respect
medical thinking is because physicians must deal with so many unknowns
and uncertainties.  My colleague, Renee Fox, noted that one of the first
things we teach a young doctor is to distinguish between what he/she
does not know and what Medicine does not know.  With diagnostic error,
we have both of those factors plus the more vexing problem of what is
knowable and what is knowable within the constraints of a specific
patient's life and setting.

Caveat: I'm not a physician, but I have spent the last 47 years focused
on research methods, measurement of error, and statistics.

Ross Koppel, Ph.D. FACMI
Sociology Dept and Sch. of Medicine
Senior Fellow, LDI, Wharton
University of Pennsylvania, Phila, PA 19104-6299
215 576 8221 C: 215 518 0134

To unsubscribe from the IMPROVEDX:
mail to:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG<mailto:SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG>
or click the following link: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>

Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>


http://LIST.IMPROVEDIAGNOSIS.ORG/<http://list.improvediagnosis.org/> (with your password)


Moderator: Lorri Zipperer Lorri at ZPM1.com<mailto:Lorri at ZPM1.com>, Communication co-chair, Society for Improving Diagnosis in Medicine

To unsubscribe from the IMPROVEDX list, click the following link:<br>
<a href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1" target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1</a>
</p>

To unsubscribe from the IMPROVEDX:
mail to:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG<mailto:SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG>
or click the following link: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>

Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>


http://LIST.IMPROVEDIAGNOSIS.ORG/<http://list.improvediagnosis.org/> (with your password)


Moderator: Lorri Zipperer Lorri at ZPM1.com<mailto:Lorri at ZPM1.com>, Communication co-chair, Society for Improving Diagnosis in Medicine

To unsubscribe from the IMPROVEDX list, click the following link:<br>
<a href="http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1" target="_blank">http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1</a>
</p>

________________________________

Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>

To unsubscribe from IMPROVEDX: click the following link:
http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1
or send email to: IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG>

Visit the searchable archives or adjust your subscription at: http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?INDEX

Moderator: Lorri Zipperer Lorri at ZPM1.com<mailto:Lorri at ZPM1.com>, Communication co-chair, Society for Improving Diagnosis in Medicine

To learn more about SIDM visit:
http://www.improvediagnosis.org/

**********************************************************
Electronic Mail is not secure, may not be read every day, and should not be used for urgent or sensitive issues

________________________________

Address messages to: IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX at LIST.IMPROVEDIAGNOSIS.ORG>

To unsubscribe from IMPROVEDX: click the following link:
http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?SUBED1=IMPROVEDX&A=1

or send email to: IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG<mailto:IMPROVEDX-SIGNOFF-REQUEST at LIST.IMPROVEDIAGNOSIS.ORG>

Visit the searchable archives or adjust your subscription at: http://list.improvediagnosis.org/scripts/wa-IMPDIAG.exe?INDEX

Moderator: Lorri Zipperer Lorri at ZPM1.com<mailto:Lorri at ZPM1.com>, Communication co-chair, Society for Improving Diagnosis in Medicine

To learn more about SIDM visit:
http://www.improvediagnosis.org/

________________________________

This e-mail message (including any attachments) is for the sole use of
the intended recipient(s) and may contain confidential and privileged
information. If the reader of this message is not the intended
recipient, you are hereby notified that any dissemination, distribution
or copying of this message (including any attachments) is strictly
prohibited.

If you have received this message in error, please contact
the sender by reply e-mail message and destroy all copies of the
original message (including attachments).








HTML Version:
URL: <../attachments/20131216/a09f90b9/attachment.html> ATTACHMENT:
Name: image001.png Type: image/png Size: 5108 bytes Desc: image001.png URL: <../attachments/20131216/a09f90b9/attachment.png>


More information about the Test mailing list