Accuracy, Overconfidence and Lack of Time

Ashley N. D. Meyer, PhD ashley.meyer2 at VA.GOV
Fri Aug 30 15:16:04 UTC 2013


First off thank you for the kind words about our study. We are glad that it has stimulated conversation.

I wanted to address a few of your comments: 

As far as the accuracy rate goes, we didn’t want all of the physicians to perform perfectly (as that would create “ceiling effects” and would make it difficult to see differences between physicians). As such, we purposely chose more difficult cases. Even the “easier” were difficult and the “more difficult” were very difficult. This may have inadvertently created “floor effects” for the more difficult cases, but the relative difference in accuracy between the easier and more difficult cases gave us the ability to compare relatively different levels of performance within each physician and how they change their confidence, which was our main aim. In everyday practice, we do not expect physicians to perform this poorly. So, hopefully, that should assuage some concerns.

In regards to your second comment on how accuracy is not improving over time- that is a more difficult question to address. Previous research has shown that physicians often make their minds up relatively early in the diagnostic process and tend to seek confirming evidence (which would not likely change their accuracy). Perhaps we are seeing some of that here. In making conclusions about the utility of expensive testing, I think I would shy away from that, as this study only looked at 4 separate cases. There are likely other diagnoses that would benefit from such testing. It might be beneficial for us to dissect the cases a little more to evaluate where in the four phases the most informative diagnostic information for each case was delivered to better get at this comment. It is definitely a good point though.

“Illusion of knowledge” in place of the term overconfidence is an interesting idea. That’s a great thought. Additional studies could look into why physicians rate their confidence as they do. Perhaps some of that is an illusion of knowledge. I would assert that there are also other factors playing into confidence judgments, including but not limited to lack of negative feedback over one’s career, self-efficacy in medicine in general, risk aversion (as many people have mentioned over the past few days), experience with similar sounding cases, and various other things.

Jason, you also brought up the issue of time in terms of it not being a barrier to ask for additional resources in our study. I think you are right- this was not a barrier. However, I do think that physicians are likely used to thinking in terms of time and efficiency, so it is possible that they are primed to think this way even in situations where time is not a factor (i.e., that barrier might exist implicitly or unconsciously). I do agree with your point though, that people often say they don’t have time to do something, when really it is just not a priority for them. Hopefully this doesn’t happen when stakes are high, but again this is an empirical question.

Lastly, the use of symptom checkers could be an interesting idea. I think in using tools like this, we just need to make sure that the tools prevent patients from falling into similar traps or using similar biases that physicians do (and frankly those that all decision-makers are susceptible to), including only looking for confirming evidence and ignoring symptoms that don’t fit (and then not reporting those to their doctors).

More information about the Test mailing list