Meta-evaluation

The procedure is direct. Domain experts label a sample of responses against the rubric criteria; each candidate grader labels the same responses; agreement between grader labels and expert labels is reported, typically as macro-F1. Grader selection then becomes an empirical result rather than a design assumption, and the reported agreement figure sets an upper bound on how much confidence any downstream score deserves.

HealthBench is the reference example in this domain. Its default grader, GPT-4.1, was chosen by meta-evaluation against physician grades, scoring macro-F1 0.709 against 0.692 for o4-mini, 0.681 for o3, 0.661 for GPT-4.1-mini, and 0.580 for GPT-4.1-nano. The benchmark also reports that GPT-4.1's agreement exceeded the average individual physician's agreement in 5 of 7 themes, which is the stated argument for grading 5,000 conversations and 48,562 criteria automatically rather than by panel.

Two limits apply to any such result. First, agreement is validated only on the distribution it was checked against. HealthBench Professional uses a different grader, GPT-5.4 at low reasoning effort with a length adjustment, on a harder and partly adversarial task set, so validation on the parent set transfers no guarantee. Second, macro-F1 against physician labels measures reproduction of physician judgment, including the places where physicians disagree with each other, which is why some suites define consensus criteria before grading.

For a buyer comparing benchmarks, the meta-evaluation section is the part that establishes whether the scores mean anything. Its absence is a stronger signal than any individual model result on the board.