Which benchmarks test AI agents in healthcare?
updated August 17, 2026
Five tracked benchmarks score agents rather than single responses: HealthAgentBench, CHI-Bench, PhysicianBench, EHR-Complex, and HealthAdminBench. They differ in environment, from terminal-based clinical artifacts and MIMIC-IV databases to real EHR APIs and computer-use administrative workflows. Most report a consistency metric alongside single-attempt success, and consistency is where scores fall furthest.
HealthAgentBench, published by Microsoft Research in July 2026, covers 54 tasks across 7 terminal-based environments built from real clinical artifacts, run as 162 trials at three attempts per task, and reports cost per task alongside mean task success rate. Its rows are agent harnesses rather than bare models, including Claude Code, Codex, and Copilot. Microsoft's own harness does not top the board it publishes, which is a useful independence signal for a vendor-run evaluation.
CHI-Bench, from actAVA, covers 75 long-horizon US healthcare operations workflows, 25 each in prior authorization, utilization management, and care management, spanning 21 healthcare applications and more than 200 MCP tools. Tasks run 60 to 80 steps across 4 to 6 stages and are scored pass@1 with a binary reward, judged by deterministic unit tests plus an LLM judge for evidence grounding, consent, and cross-stage consistency. It was released May 20, 2026 and updated August 12, 2026 across 45 harness configurations, and it accepts community submissions, so rows mix author-run and submitted results.
PhysicianBench and EHR-Complex are paper-based rather than live boards. PhysicianBench covers 100 real-world clinical tasks across 21 specialties with 670 structured checkpoints and roughly 27 tool calls per task, verified by execution against actual EHR systems through standard commercial APIs, reported as pass@1 over three runs with Pass^3 also given. EHR-Complex covers about 52,000 tasks with a 3,915-task test set over 365,000 patients, 31 tables, and more than 500 million records, requiring SQL and Python over MIMIC-IV across six clinical intents, scored by exact match.
HealthAdminBench, built by Kinetic Systems with Stanford Hospital domain experts, scores computer-use agents on 135 healthcare administration tasks broken into 1,698 rubric-scored subtasks covering prior authorizations, denial appeals, and durable medical equipment ordering. Success is end to end: every subtask in a task must complete. It is independent of the model vendors, though published by a company selling healthcare administration automation, and it carries no stated refresh cadence.
Agentic capability also enters the ARISE MAST composite, which folds MedAgentBench v2 and PhysicianBench into a percentage score across 11 models alongside diagnostic, safety, and multimodal components. MAST is marked as a preview, last updated August 15, 2026, and publishes component-level breakdowns only for First, Do NOHARM v2. Across this category, the number worth reading is consistency rather than single-attempt success: EHR-Complex reports consistency below 50 percent at Pass^4 for nearly every model, PhysicianBench shows several agents near 1 percent on Pass^3, and no CHI-Bench agent held above 20 percent across three identical runs at launch. Current standings for each are on clinicalbenchmarks.ai.