Overview
The Limits of Expertise · 01
In a lung ultrasound study we published in JMIR, six experts, four of them fellowship-trained emergency physicians, sorted 198 test clips into three B-line categories. They agreed unanimously on 104. On the other 94, almost half the set, at least one expert disagreed, and the 195 training clips split almost the same way. These were experienced readers working inside their own field.
The usual way to handle disagreement like this is to recruit enough credentialed readers to form a majority, take the vote, and keep one label per case. The split that produced each label is gone by the time the dataset ships, even though it's the best evidence available about where the reference standard is solid and where it isn't. Our view is that disagreement belongs in the reference standard itself, because it shows which labels are uncertain, which label definitions are doing less work than they appear to, and when the reference itself deserves a second look.
This is the first of three posts in The Limits of Expertise, a series on what goes into a clinical reference standard and where expert judgment needs measurement around it. Part 01 looks at disagreement between specialists, Part 02 at what credentials do and don't predict about a reader, and Part 03 at reader time, the constraint that shapes every study plan.
A majority label hides how certain it is
The six experts reached at least a two-thirds majority on 174 of the 198 test clips. That leaves 70 clips where the majority label came with one or two experts dissenting, and 24 where no label reached two-thirds at all. The study's reference standard took the majority on every clip and broke ties at random. Once labels are flattened into a single column, a clip settled by a random tiebreak looks exactly like one that all six experts agreed on.
The certainty behind a label is measurable, and it predicts accuracy. In the same study, when at least 80 percent of crowd opinions on a clip agreed, the crowd label matched the expert reference 96 percent of the time. Expert agreement was 89 percent on clips the crowd labeled correctly and 60 percent on clips it got wrong, and clip by clip, crowd agreement tracked expert agreement (r = 0.70). The same cases were hard for both groups, so case-level agreement says something real about how reliable each label is.
Disagreement points at definitions
In our experience, the hardest calls cluster where a definition draws a line. Discrete B-lines sit between two other categories, no B-lines and confluent, so every discrete label depends on where two boundaries are drawn, and Part 02 shows how widely individual experts varied on that category. In our MICCAI 2024 work on B-line segmentation, telling single B-lines apart from merged ones remained a source of ambiguity.
That's why we treat labeling instructions as part of the study method. They take longer to get right than teams expect, and we end up building a living FAQ and ontology for most specialist tasks, covering which findings qualify, what context readers may use, where a boundary sits, and what happens when a finding can't be assessed. The questions readers ask in training are the most reliable map of those grey areas, and the answers belong in the study record alongside the labels.
The reference can be wrong too
Agreement data is also how a questionable reference gets caught. Our QA process tracks two scores. Agreement measures how often labelers give the same answer as each other. Difficulty measures the share of qualified reads that differ from a case's correct label. High agreement paired with high difficulty is the signature of a questionable reference: readers converging on one answer while the reference says another. Labelers can also flag inappropriate, irrelevant, or misidentified images, so problem cases surface during labeling.
Gold standards are strongest when they come from several experts, because the experts' own disagreement rate becomes the benchmark everyone else is measured against.
Why this matters more as models improve
When a model performs well below expert level, label noise in the reference standard matters relatively little. As models close that gap, uncertainty in the reference becomes a larger share of what an evaluation can resolve, and contested cases start to decide the headline number. A model graded against a majority label on a 4-to-2 split is being graded against a label a third of the experts rejected.
Regulators ask how ground truth was established for the same reason. A reference standard that records where readers disagreed, how those cases were resolved, and what instructions readers were working from can answer that question with data. One that records only the final label can't.
These rates describe this task, and another finding, modality, or label set will have its own. In customer studies, the manufacturer owns the validation methodology and our role covers the labeling work its protocol defines. Within that scope, we think reference standards should keep their disagreement data and use it, both to decide which cases get adjudicated and to show how much weight each label can bear.
Part 02 looks at the other side of the same problem. If specialists split this often, the next question is whether the readers producing a reference standard can be measured, and what a credential does and doesn't tell you about them.
