Model grader

Rubric benchmarks in health require a judgment per criterion, and there are far more criteria than human graders can process repeatedly. HealthBench uses GPT-4.1 as its default grader across 48,562 criteria. HealthBench Professional uses GPT-5.4 at low reasoning effort together with a length adjustment. CHI-Bench splits the work, using deterministic unit tests for what can be checked mechanically and an LLM judge for evidence grounding, consent, and cross-stage consistency.

A grader is selected by meta-evaluation against human grades rather than by assumption. OpenAI chose GPT-4.1 for HealthBench on macro-F1 agreement with physician grades: 0.709 for GPT-4.1, against 0.692 for o4-mini, 0.681 for o3, 0.661 for GPT-4.1-mini, and 0.580 for GPT-4.1-nano. On the same measure, GPT-4.1 exceeded the average individual physician's agreement in five of the seven HealthBench themes.

Grader choice is part of the result rather than a footnote beneath it. Two runs of the same benchmark under different graders are different measurements, which is one reason rows on HealthBench from different sources do not line up. Some benchmarks reduce single-grader dependence structurally: Health Optimization Bench grades blind with a panel of three model families and never allows the family that authored a task to grade it.

Grader variance is small but not zero. Repeating HealthBench 16 times produced a standard deviation of about 0.002 on the overall score. That figure sets the floor for how small a difference between two models can be reported as real, and it is why a board whose leaders sit within a fraction of a point of each other has stopped discriminating. Grader configurations are recorded alongside current scores at clinicalbenchmarks.ai.