Grader bias

In rubric benchmarks the grader decides, criterion by criterion, whether a response earned credit. Its errors therefore propagate into every score the benchmark reports. Bias differs from noise in a way that matters for ranking: noise widens intervals and washes out across repeats, while bias shifts scores in a consistent direction and survives averaging.

Verbosity is the best-documented case. Unadjusted rubric scoring rewards a response that enumerates more material, whether or not the extra material helps the reader, because more text collides with more criteria. HealthBench's length adjustment penalizes about 2.99 points per 500 characters beyond 2,000 characters, and the corresponding adjustment on HealthBench Professional is 1.47. Whether a published score is length-adjusted changes both its value and its comparability. See length adjustment.

Self-preference is the second case: a grader tends to favor responses resembling its own family's output. Health Optimization Bench is structured around this risk. Every task is written against a primary source, audited by model families that did not author it, and scored blind by a panel of independent families, with the authoring family never grading its own task. The board publishes 95 percent bootstrap confidence intervals beside every score.

Bias is measured, not assumed away. HealthBench selected GPT-4.1 as its default grader after checking agreement with physician grades: macro-F1 0.709, against 0.692 for o4-mini, 0.681 for o3, 0.661 for GPT-4.1-mini, and 0.580 for GPT-4.1-nano. That procedure is meta-evaluation, and a benchmark that reports no such check gives no evidence about its grader's bias.