Disease Prevalence Effects on Dermatological AI Decisions

Imagine that you are a dermatologist. You spend the morning seeing patients who have been referred to you for suspicion of skin cancer. Many of them do, in fact, have skin lesions that require treatment. We would say that disease "prevalence" was high in this set of patients.

‍

Suppose that you next spend the afternoon giving annual screening exams to members of the general population. Here disease prevalence will be low. Would your morning’s work influence your decisions about patients in the afternoon?

‍

In collaboration with Centaur Labs, Jeremy M. Wolfe, PhD, Professor of Ophthalmology & Radiology at Harvard Medical School, conducted a study published this month in Cognitive Research: Principles and Implications (CRPI) that tackled this question. We know from other contexts that recent history can influence current decisions, and we know that target prevalence has an impact on decisions.

‍

In this study, Centaur Labs collected decisions about skin lesions from individuals with varying degrees of expertise using our proprietary medical imaging labeling app, DiagnosUs. Over 300,000 trials were collected in DiagnosUs over five days, with 803 participants in the study. This allowed the researcher to examine the effects of feedback history and prevalence in a single study.

‍

This research finds that feedback educates observers, causing them to become more liberal when targets (diagnoses of melanoma) have been relatively common and more conservative when those targets are rare. The effects of a block of trials with feedback can last for days, with those effects showing up when the observer takes up a similar task again. It may be possible to use the educational effects of feedback when it is desirable to shift an observer's criterion, especially if the subsequent task does not involve reliable feedback.

Schedule a demo with Centaur.ai
‍

Read the full study here:
‍Cognitive Research: Principles and Implications (CRPI)