pass@k

Agentic health benchmarks run each task more than once because a single run is a sample rather than a measurement. HealthAgentBench runs 54 tasks three times each, for 162 trials. PhysicianBench reports pass@1 over three independent runs and also reports Pass^3 across the same runs. EHR-Complex reports consistency at Pass^4.

The two statistics diverge sharply in this domain. On PhysicianBench, which covers 100 real clinical tasks across 21 specialties with 670 structured checkpoints and roughly 27 tool calls per task, the lower half of the table falls to near 1 percent on Pass^3. EHR-Complex reports consistency below 50 percent at Pass^4 for nearly every model evaluated. CHI-Bench led its May 2026 launch report with reliability rather than capability: no agent stayed above 20 percent across three identical runs.

The gap matters for procurement because a deployed workflow behaves more like Pass^k than pass@1. A prior authorization that completes once in three attempts is a failed workflow, not a partial success. CHI-Bench scores pass@1 with a binary 0 or 1 reward on workflows of 60 to 80 steps across four to six stages, so even its single-attempt headline assumes the whole chain completed.

Reported k values are not comparable across benchmarks, and neither are the task lengths behind them. A pass@1 figure on a short task and a pass@1 figure on a 60 step workflow describe different things at the same nominal value. Per-benchmark definitions sit with the current scores at clinicalbenchmarks.ai; see also agentic benchmark.