Overview
The Limits of Preference · 04
In one group of annotators, ChatGPT beat Gemini in 81.8 percent of head-to-head comparisons. In another group from the same study, Gemini beat ChatGPT 84.2 percent of the time. Both results came from questions held out of the process used to identify the groups.
The models and evaluation task remained unchanged, what differed was the group of people evaluating the responses. Depending on which annotators contributed to the comparison, the resulting consumer-preference leaderboard told a different story.
Across the first three posts of our Limits of Preference series, we've been investigating what conventional preference evaluation can miss. In Post 01, we tested whether answer features associated with preference were attributable to those features or to the models producing them. In Post 02, we introduced controlled flaws into health responses and found that annotators penalized dismissive language more consistently than factual errors. In Post 03, we measured how changes in empathy and clarity affected preference, finding that comparable declines in rubric scores across these dimensions produced substantially different preference penalties.
Together, these experiments show why health AI evaluation cannot rely on preference rankings alone. Understanding model performance requires separating what people prefer from the clinical qualities that matter for safe and appropriate responses. Our fourth analysis examines another part of that measurement, the people making the judgments.
When a leaderboard reports which model humans prefer, whose preferences does that result actually represent?
Methods
Our original study collected 26,780 preference judgments from 132 annotators across 712 consumer health questions. Four language models answered each health query, producing 2,848 model responses. Annotators compared pairs of answers and selected which one they would rather receive.
Aggregating those choices produced an overall consumer-preference ranking: Google's Gemini 3.5 Flash first, OpenAI's GPT 5.5 Instant (served through chat-latest) second, Anthropic's Claude Sonnet 4.6 third, and Meta's Llama 4 Maverick (17B) fourth. For this particular analysis, we wanted to understand whether that aggregated preference ranking concealed meaningful differences between groups of annotators.
Rather than grouping people according to which model they preferred, we examined the characteristics of the answers they tended to select.
For each annotator, we measured how often they chose a longer response, an answer with more section headers, one containing a disclaimer, or one using more hedging or questions, we also measured their typical reading time.
Each selection rate was calculated only from comparisons where the two answers differed on that characteristic. Model identity and our own response quality scores were not used to form the groups.
To test whether these differences extended beyond the questions used to identify them, we divided the 712 questions into two halves.
We used judgments from the first half to group annotators according to their observed behavior, we then calculated each group's model rankings using only judgments from the second half, which had played no role in forming the groups.
We repeated the process in reverse, using the second half for grouping and the first half for evaluation.
Both directions produced the same broad preference profiles and the same models at the top of each group's ranking.
The analysis identified three distinct patterns among annotators with enough judgments to be grouped.
One group tended to select plainer answers with fewer structural elements and spent relatively little time reading. Another group preferred longer answers and was more accepting of disclaimers. A third preferred structured answers without disclaimers and spent substantially longer reading.
These patterns are not fixed personality types. Annotators can fall between groups, and preferences for length, formatting, and disclaimers can overlap, but the differences were consistent enough to ask a more consequential question. If each group evaluated the same set of models independently, would they produce the same leaderboard?
Rankings
The differences were substantial.
Group A, which favored plainer responses and read relatively quickly, ranked ChatGPT first. In direct comparisons with Gemini, this group selected ChatGPT 81.8 percent of the time. The reverse split produced a similar result at 79.1 percent.
Group B, which preferred longer responses and was more accepting of disclaimers, put Llama level with Gemini at the top. Their direct matchup was nearly even, splitting 51.1 to 48.9 percent. Llama also beat ChatGPT in 66.3 percent of comparisons.
Group C, which favored structured responses without disclaimers and spent longer reading, ranked Gemini first. This group selected Gemini over ChatGPT 84.2 percent of the time, rising to 84.7 percent in the reverse split.
The contrast between Groups A and C is particularly revealing, one group overwhelmingly preferred ChatGPT to Gemini. The other overwhelmingly preferred Gemini to ChatGPT.
The overall leaderboard placed Gemini first and ChatGPT second, but that aggregate result did not describe how either group judged the same model matchup. Llama showed an even more pronounced difference under a simpler analysis.
When we grouped annotators into just two groups, Llama moved from last in the overall leaderboard to sharing the top position with Gemini and ChatGPT among the plainer, quicker-reading annotators. Among the structured, slower-reading group, Llama remained last, losing to Gemini in 94.2 percent of their direct comparisons.
The conclusion did not depend on choosing exactly three groups. The broader finding remained when we divided the crowd into two.
Implications
It would be easy to explain these results as people having different tastes. That is true in a general sense, but the evaluation problem is more specific.
A preference leaderboard combines judgments into an aggregate score. That score describes the preferences represented in the collected data, but it can conceal substantial disagreement between groups of annotators.
In our overall ranking, Gemini performed best. Yet a panel consisting of annotators with Group A's observed preferences would have placed ChatGPT first. One with Group B's preferences would have placed Llama alongside Gemini.
Each ranking describes the observed preferences of a particular group. None establishes which model is more medically accurate or clinically safe. Those questions require separate evaluation criteria and appropriately qualified reviewers.
This distinction becomes important when a preference score is used to compare models, select training data, or guide model development. A team may believe it is optimizing for human preference when the measurement is more specifically capturing the preferences of the annotators it recruited.
The pool of reviewers therefore helps define the evaluation itself.
These groups describe behavior within our consumer-preference panel, not professional backgrounds. We did not use self-reported credentials to define the groups or make claims about clinicians' preferences.
It also does not mean that reading more slowly makes someone a better evaluator. Reading pace was unrelated to accuracy on our quality-check questions, and dividing annotators by reading speed alone did not reproduce the different winners.
What mattered in this analysis was the combination of what annotators tended to choose and how they approached the comparisons.
Design
A preference score compresses many individual judgments into a single result. That makes comparisons easier, but it can also conceal disagreement that materially changes how a model's performance should be interpreted.
Our findings show that the composition of an annotator panel is part of the measurement itself. Different groups within the same evaluation produced different model rankings, even when those rankings were tested on questions the grouping process had never seen.
For teams developing health AI, this makes the design of the evaluation population as important as the design of the benchmark. Reviewer selection should reflect the intended use case, and the evaluation should preserve enough information to examine how preferences vary across relevant groups.
But selecting the right reviewers is only one part of the problem.
A model can produce an answer that users prefer while still omitting clinically necessary guidance. An answer can receive favorable ratings for empathy or clarity without meeting the standards required for medical accuracy or safety. And an aggregate preference ranking can conceal meaningful differences between the people whose judgments produced it.
These are separate measurement problems, and they require separate evaluation criteria.
In CLEAR-Bench, we approached this by evaluating consumer preference alongside independent clinical assessments of factual accuracy, urgency, and safety. The resulting data allowed us to examine not only which responses people preferred, but also where those preferences diverged from concerns raised by screened clinicians.
A more complete health AI evaluation should preserve those distinctions. It should establish the intended population, define clinically meaningful criteria, use appropriately qualified reviewers for specialist judgments, and retain individual decisions so disagreement can be examined and traced back to the underlying cases.
Where a task permits objective quality measurement, reviewer performance can also inform how evidence is combined. Where the target is subjective preference, differences between groups need to remain visible rather than being treated as measurement error.
The goal is not to eliminate disagreement or produce a single universal ranking. It is to build an evaluation that makes the sources of disagreement, the limitations of each measurement, and the clinical implications of model behavior understandable.
Our overall consumer-preference leaderboard summarized the choices collected from our panel. This analysis shows why that summary should not be mistaken for a universal account of human preference, much less a complete measure of medical model quality.
Who participates in an evaluation helps determine what the result means. For health AI, that decision belongs at the center of the evaluation design.