Multi-turn benchmark

Each item supplies a conversation prefix; the model produces the next response; the response is graded against criteria written for that specific conversation. Static versions replay fixed prefixes, which makes runs reproducible. Dynamic versions simulate a user that reacts to what the model just said, which resists memorization but makes exact replication impossible.

HealthBench (OpenAI, May 2025) is the standard reference. It contains 5,000 health conversations with 48,562 rubric criteria written by 262 physicians across 60 countries, 26 specialties, and 49 languages, with a median of 11 criteria per conversation and a range of 2 to 48. Criteria carry weights from -10 to +10, and negative criteria subtract for harmful content. An example score is earned points divided by maximum positive points, so a single example can score below zero; the overall score is the mean clipped to the 0 to 1 range. Rerun variability on a fixed set is small, with a standard deviation of about 0.002 over 16 repeats.

HealthBench Hard is the derived subset: the 1,000 conversations where five frontier models scored lowest at the May 2025 release, after removing the roughly 1.5 percent where no model scored positive. At that release, o3 led it at 0.320 against 0.60 on the full set, and GPT-3.5 Turbo, GPT-4o (August 2024), and Llama 4 Maverick scored 0.00. Current standings are on healthbenchhard.ai.

OpenAI's Dynamic Mental Health Evaluations show the dynamic form, simulating users whose conversations evolve in response to model outputs across mental health, emotional reliance, and self-harm response quality. Multi-turn benchmarks differ from agentic benchmarks in that the environment is the interlocutor: there is no tool surface and no external state to verify.