Overview
The question we’re testing
At Centaur, we're developing an evaluation of how healthcare AI assistants respond as patients add or correct information over several turns in a conversation.
The question behind it is a simple one. If two patients tell a model the same things in a different order, does it give them the same advice?
When the same facts arrive in a different order
The full sequence of patient messages, model responses, and changes in context is the conversational trajectory. Two conversations can share an endpoint, meaning the model has been told exactly the same information over the course of the conversation, and still differ completely in trajectory. Our evaluation holds the information constant and varies only the trajectory.
Stability means giving clinically consistent advice once the model has the same relevant patient information. The recommendation should not depend on the order in which the facts arrive, whether one of them was corrected along the way, or how they are split across turns. Degradation caused by a presentation-only change is a defect.
Stability is the property we're testing.
Updating, retraction, and persistence
Three behaviors determine whether a model achieves it, and each can fail on its own:
Updating means revising a recommendation when new patient information changes what advice is appropriate. The model's next response should account for the clinical implications of that information.
Retraction means withdrawing or correcting earlier advice that no longer applies, so that a recommendation which has since become unsafe is not left standing.
Persistence means continuing to use relevant patient information in later responses. The model should still account for an established fact when the patient returns to the question, even after the conversation has moved to another topic.
Each of these is graded across the whole conversation, not on the final reply alone.
Where the evaluation stands
We’re developing this evaluation across many clinical contexts and patient scenarios. Clinical validation is underway, and we’ll publish the results once that work is complete.
