keystone-bench
A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBen
What is keystone-bench?
A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.
Overview of keystone-bench
What it is
Keystone-Bench is a clinical decision-evidence benchmark. It is built on HealthBench. The benchmark tests whether a chat assistant's action changes when a single decisive fact in a clinical conversation is removed, replaced, or contradicted. It also measures whether the rubric used to grade the assistant remains applicable after such an edit.
Key features
- 7,419 twins over 1,260 HealthBench conversations, including 2,861 multi-turn items.
- Nine families of edits: missing_evidence, conflicting_evidence, demographic_shift, salient_distractor, alternative_evidence, demographic_control, buried_red_flag, missing_evidence_early, and alternative_evidence_hard.
- Decision–evidence layer: per source a decision frame; per twin one of five evidence states with acceptable actions, forbidden actions, and decisive questions.
- Paraphrase control for 1,132 sources: reworded, nothing added or removed.
- Multiple layers and splits: all, core, strict, primary, quick; sources split 80/20 into dev and test.
- Reference results for five assistants, grader validity checks, and auditable SHA-256 per file.
Use cases
Evaluating clinical chat assistants on their ability to correctly change or maintain actions when a single fact is altered. Testing the robustness of grading rubrics against edits that change the evidence state. Auditing whether an assistant's reply is driven by the evidence or by superficial cues.
Getting started
pip install keystone-benchkeystone buildSee the repository README for full instructions.
License and pricing
The license is not stated. The project is open source.
Based on the project's public README and repository metadata.