How are medical AI benchmarks graded?

Rubric grading is the dominant method for open-ended clinical text. HealthBench carries 48,562 criteria written by 262 physicians across 60 countries, 26 specialties, and 49 languages, with a median of 11 criteria per conversation and a range of 2 to 48. Criteria carry weights from -10 to +10, and negative criteria subtract points for harmful content. The example score is earned points divided by the maximum available positive points, so an individual example can be negative; the overall score is the mean across examples, clipped to a 0 to 1 range.

Under rubric grading, the grader model is itself an object of evaluation. GPT-4.1 was selected as the HealthBench default by meta-evaluation against physician grades, reaching macro-F1 0.709 against 0.692 for o4-mini, 0.681 for o3, 0.661 for GPT-4.1-mini, and 0.580 for GPT-4.1-nano, and exceeding the average physician's agreement in 5 of 7 themes. HealthBench Professional uses a three-stage rubric adjudication with three or more physicians per criterion, grades with GPT-5.4 at low reasoning effort, and applies a length adjustment that penalizes verbosity.

Grader independence is handled differently across boards. Health Optimization Bench writes every task against a primary source, audits it with model families that did not author it, and scores blind with a panel of independent families, so the authoring family never grades its own task; 95 percent bootstrap confidence intervals accompany every published score. Where a single grader family both authors and scores, that dependency belongs in the reading of the result.

Agentic benchmarks replace judgment with verification wherever the task allows it. CHI-Bench scores pass@1 with a binary reward from deterministic unit tests, adding an LLM judge only for evidence grounding, consent, and cross-stage consistency across its 60 to 80 step workflows. PhysicianBench checks 670 structured checkpoints against real EHR systems through standard commercial APIs, reporting pass@1 over three independent runs alongside Pass^3. EHR-Complex uses exact-match accuracy over MIMIC-IV queries, and HealthAgentBench reports mean task success across 162 trials at three attempts per task.

Human annotation supplies ground truth where neither a rubric nor a unit test is sufficient. First, Do NOHARM v2 rests on 12,747 expert annotations of 4,249 management options across 1,100 consultation cases, scoring frequency and severity of potentially harmful errors. MedCode grades ICD-10-CM assignment for full hospital stays against ground truth double-annotated by certified professional coders across 2,755 records. Which method a benchmark uses, and which grader version produced a given row, is recorded per benchmark on clinicalbenchmarks.ai.