Do benchmark scores predict real clinical performance?
updated August 17, 2026
No tracked healthcare AI benchmark establishes that its scores predict clinical outcomes. Each measures performance on a defined task set, under a stated grading protocol, usually against physician-written rubrics rather than against patient results. Scores are best read as evidence about a specific construct, and they should be matched to the workflow a buyer intends to deploy.
Rubric benchmarks score text against criteria physicians wrote, applied by a model grader. HealthBench uses 48,562 criteria written by 262 physicians across 60 countries, 26 specialties, and 49 languages, with a median of 11 criteria per conversation and weights from -10 to +10, where negative criteria subtract for harmful content. What the score measures is agreement with those criteria. It is not a measurement of what happens when a clinician acts on the response.
The grading layer has its own measured error. HealthBench's default grader, GPT-4.1, was selected by meta-evaluation against physician grades, reaching macro-F1 0.709 against 0.692 for o4-mini, 0.681 for o3, 0.661 for GPT-4.1-mini, and 0.580 for GPT-4.1-nano, and exceeding the average physician's agreement in 5 of 7 themes. Agreement above 0.7 is strong for an automated grader and still leaves a gap. Any score inherits that gap.
Physician baselines make the scale interpretable without making it predictive. On the parent HealthBench, physicians writing unaided scored 0.13, physicians with September 2024 model references scored 0.31, and physicians with April 2025 model references scored 0.48, no longer improving on the references, which scored 0.49 alone. On HealthBench Professional, physician-written responses score 0.437. These figures anchor a score against physician-produced text on the same rubrics, which is a narrower statement than clinical equivalence.
Deployment introduces variables that model-level benchmarks do not score. HealthAgentBench evaluates agent harnesses end to end rather than bare models, and CHI-Bench's results across 45 harness configurations indicate that harness choice matters as much as model choice. Repeat reliability separates further: EHR-Complex reports consistency below 50 percent at Pass^4 for nearly all models, and no CHI-Bench agent stayed above 20 percent across three identical runs at launch. First, Do NOHARM v1 found errors of omission behind more than 80 percent of severe errors, a failure mode that rubric scoring on response content captures only partially.
For procurement, the usable approach is to match construct to workflow: ICD-10 coding accuracy from MedCode, documentation quality from MedScribe, consultation safety from First, Do NOHARM, and agentic workflow completion from the agent benchmarks. Composite indexes need more care, since the Artificial Analysis Healthcare and Medical Index is a weighted composite of general-purpose benchmarks tilted toward health-relevant slices rather than purpose-built clinical tasks. Current per-benchmark scores are published on clinicalbenchmarks.ai.