Eval harness

The settings that move scores are identifiable and few: reasoning effort, grader model and version, whether a length adjustment is applied, the tool surface exposed to the model, retry and attempt policy, and how per-item scores are clipped and averaged. A published score without its harness configuration is not reproducible and not comparable to any other score.

HealthBench grades with GPT-4.1 by default. HealthBench Professional grades with GPT-5.4 at low reasoning effort and applies a length adjustment that penalizes verbosity. Those are different measurement instruments applied to different sets, and their numbers do not convert. The cross-source problem is visible on HealthBench itself: Anthropic reports a raw score under its own protocol, OpenAI leads with length-adjusted numbers at maximum reasoning effort and reports production Instant settings separately, and Baichuan ran its competitors itself. See vendor-reported score and independent run.

In agentic evaluation the harness stops being a wrapper and becomes part of the system under test. HealthAgentBench's rows are agent harnesses such as Claude Code, Codex, and Copilot rather than bare models, and Microsoft Research runs the board itself. CHI-Bench's August 12, 2026 update spans 45 harness configurations, with its maintainers reporting that harness choice matters as much as model choice.

For evaluation buyers, the practical rule is to record the harness configuration next to every score in a comparison table, and to treat rows whose configuration is unstated as unusable for ranking. Per-benchmark pages that state the basis of each row are at clinicalbenchmarks.ai.