Beyond scripted conversations
Many recent health AI benchmarks fix the evidence available to a model before the evaluation starts. A model gets a clinical question, a vignette, or a conversation that has already been written, produces an answer, and that answer is scored for things like accuracy, appropriate next steps, and unsafe advice. That works well when the goal is to evaluate a response to a known set of facts, but it is an imperfect approximation of a real patient interaction. In an actual conversation, what the model learns next depends partly on what it just asked, misunderstood, ignored, or decided was important.
Multi-turn benchmarks get closer by spreading information across several conversation turns, but a longer conversation is not necessarily a dynamic one. If the full dialogue is written before the model runs, every model still receives essentially the same evidence in the same order, regardless of how it behaves. A model that asks about a recent surgery immediately and a model that never asks about it may both eventually receive that fact simply because it was scripted to appear on turn four. That is useful for testing memory, contradiction handling, and information order, but it cannot tell us whether the model would have actually surfaced the information it needed.
One patient, different paths
The technical step is to treat the patient as part of the evaluation environment rather than as a transcript. Each case can begin with a fixed underlying patient state: demographics, presenting complaint, medical history, symptoms, relevant negatives, persona, and rules describing which information is volunteered versus disclosed only when the model asks for it. A separate synthetic patient model then expresses that state during the conversation, while the health model being evaluated remains an entirely different system. The patient can respond differently depending on what the model does, but it stays grounded in the same case rather than inventing new pathology or conveniently revealing the diagnosis.
That gives the evaluation something important: controlled variability. Two models can now have very different conversations with the same underlying patient, while the clinical state, disclosure rules, and scoring criteria remain fixed. Reproducibility no longer has to mean replaying an identical transcript; it can mean holding the environment constant and observing how different systems behave inside it. The transcript becomes an outcome of the evaluation rather than the evaluation artifact itself, and because each patient is defined as a structured state rather than a recorded encounter, this approach also reaches cases that are hard to source from real data. Rare presentations, high-acuity scenarios, and populations that are expensive or slow to collect at scale can be specified directly by clinicians and run against any model.
Scoring the full conversation
That changes the unit of analysis from a completion to a trajectory. A final answer can tell us whether the model eventually reached a reasonable conclusion, but it can hide a lot of what happened before that point: whether the model elicited the important risk factor, updated after a correction, recovered from a bad assumption, contradicted itself, or gave unsafe advice halfway through the encounter. At a technical level, the object being evaluated starts to look less like a single response and more like a policy operating over a stateful environment. That makes it possible to score how the model navigated the sequence of states that led there.
How the evaluation works
Our architecture separates dynamic evaluation into three components: a synthetic patient agent, the health model under test, and an independent clinical scoring layer. Each evaluation starts with a defined patient state that includes medical history, risk factors, presenting symptoms, and rules for which information the patient volunteers versus what they disclose only when asked. The patient agent receives that information, but not the underlying diagnosis, and responds to the model under test turn by turn. That means two models can encounter different information and take entirely different paths through the same clinical case, depending on the questions they ask and the decisions they make. We then evaluate those trajectories against clinician-defined criteria for essential questions, diagnostic reasoning, red flags, unsafe recommendations, and appropriate disposition, allowing us to measure not just whether a model reached the correct conclusion, but how it got there.
Where it fits
Static benchmarks still matter, and fixed multi-turn evaluations remain valuable when the goal is controlled testing of a specific capability. But as health models move into longer conversations, copilots, and more agentic workflows, some of the failures that matter only appear once the environment can react to the model. Dynamic evaluations give us a way to test that behavior without giving up control of the underlying case.
