Agentic benchmark

updated August 17, 2026

An agentic benchmark scores completion of multi-step tasks in an environment with tools and persistent state, rather than the quality of a single response. Success is usually verified by inspecting the resulting state rather than by grading prose.

The structure has four parts: a task specification, an environment the system can change, a tool surface, and a verification method. Verification tends to be execution-grounded, using unit tests, structured checkpoints, or database queries, which removes the grader-agreement problem that rubric benchmarks carry but replaces it with sensitivity to environment design and tool availability.

Healthcare examples differ in what they place under test. HealthAgentBench (Microsoft Research, July 2026) runs 54 tasks across 7 terminal-based environments built from real clinical artifacts, with 162 trials at three attempts per task, and its rows are agent harnesses rather than bare models. PhysicianBench covers 100 real-world clinical tasks across 21 specialties with 670 structured checkpoints and roughly 27 tool calls per task, verified against actual EHR systems through commercial APIs. EHR-Complex scores agentic reasoning over MIMIC-IV with a 3,915-task test set drawn from about 52,000 tasks, using exact match on answers produced through SQL and Python. CHI-Bench covers 75 long-horizon US healthcare operations workflows of 60 to 80 steps across 4 to 6 stages, using 200+ MCP tools and judging with deterministic unit tests plus an LLM judge for evidence grounding and cross-stage consistency.

Single-attempt scores overstate agentic performance, and the boards say so. CHI-Bench's launch report led with reliability rather than capability: no agent stayed above 20 percent across three identical runs. EHR-Complex reports consistency below 50 percent at Pass^4 for nearly every model. HealthAdminBench requires every subtask to succeed for a task to count, across 135 tasks and 1,698 rubric-scored subtasks. Repeated-run metrics are covered under pass@k.

Because the scaffold executes the task, the eval harness is part of the system under test rather than a wrapper around it. CHI-Bench's August 12, 2026 update spans 45 harness configurations, and its maintainers report that harness choice matters as much as model choice.