Red-teaming

Red-teaming inverts the sampling logic of a benchmark. A representative set estimates how a model behaves on the traffic it will see. A red-teamed set estimates how it behaves at the edge, where errors are most expensive. The two answer different questions and cannot be averaged into one number without losing the distinction.

HealthBench Professional (OpenAI, April 2026) shows the method applied inside a task set drawn from real use. Its 525 tasks were selected by physicians from 15,079 real clinician conversations, with difficulty enriched roughly 3.5 times over the source distribution and about one third of the set adversarial. The rubrics were written by 190 physicians across 50 countries and 26 specialties, through a three-stage adjudication with at least three physicians per criterion. WHBench takes the constructed route instead: 47 expert-crafted scenarios aimed at named failure modes including outdated guidelines, unsafe omissions, dosing errors, and equity blind spots.

Some red-teaming is dynamic rather than fixed. OpenAI's Dynamic Mental Health Evaluations (August 2026) simulate users whose conversations evolve in response to model outputs instead of following scripts, covering mental health, emotional reliance, and self-harm response quality. Dynamic simulation resists memorization of a static set, but this particular eval is run by the vendor on its own models, is not independently runnable, and OpenAI notes its error rates are not representative of average production traffic. See vendor-reported score.

For procurement, a red-teamed score is not a deployment failure rate. It is conditional on an input distribution chosen to be hard, so it bounds behavior under stress rather than describing behavior under load. Buyers reading a single number should confirm what fraction of the set is adversarial before comparing it to anything. Current standings on the tracked boards are at clinicalbenchmarks.ai.