Bootstrap confidence interval

The computation draws n items with replacement from the n scored items, recomputes the aggregate score, repeats that several thousand times, and takes the 2.5th and 97.5th percentiles of the resulting distribution for a 95 percent interval. Width falls as the item count rises and as per-item variance falls. It requires no distributional assumption, which is why it is the default for rubric and task-success metrics.

Healthcare eval sets are small enough that this matters at every reading. HealthBench Professional has 525 tasks, MedScribe has 100 rubric-scored cases, HealthAgentBench has 54 tasks, and WHBench has 47 scenarios. At those sizes, intervals are routinely wider than the gaps between adjacent rows on a board, so rank order can invert on a different sample of the same task population without any model changing.

Health Optimization Bench publishes 95 percent bootstrap intervals beside every score on its board, computed over the 89 released tasks in the v1 evidence suite, alongside its cross-family authoring and blind three-family panel grading. See healthoptimizationbench.com. Where a board omits intervals, small differences between rows carry no stated uncertainty and should not be reported as differences.

Three sources of variance are separate and should not be conflated. Item sampling variance is what the bootstrap estimates. Rerun variance is what repeated runs on a fixed set measure, and it can be much smaller: HealthBench reports a standard deviation of about 0.002 over 16 repeats, which says nothing about how the score would move on a different sample of conversations. Agentic suites add attempt-to-attempt variance, reported separately as pass@k and Pass^k consistency.