Adversarial example

Benchmarks drawn from representative traffic measure average behavior. Adversarial items measure the tail, which is where clinical harm sits. About one third of the 525 tasks in HealthBench Professional are adversarial, alongside a difficulty enrichment of roughly 3.5 times relative to the source conversations they were selected from.

The failure modes are documented rather than hypothetical. WHBench, 47 expert-crafted women's health scenarios graded on a 23-criterion rubric, targets outdated guidelines, unsafe omissions, dosing errors, and equity blind spots. First, Do NOHARM found in its first version that direct application of model consultation recommendations carried potential for severe harm in up to 24.6 percent of cases, and that errors of omission accounted for more than 80 percent of the severe errors. Omission is the failure a coverage-oriented rubric is least likely to catch on its own, which is why rubrics carry negative items: on HealthBench, criteria are weighted from -10 to +10, and negative criteria subtract when the harmful content they describe appears.

Adversarial items can also be generated dynamically. OpenAI's dynamic mental health evaluations run multi-turn simulated users whose conversations evolve in response to model outputs rather than following fixed scripts, covering emotional reliance and self-harm response quality. Dynamic generation limits contamination, since no fixed item exists to memorize, and it costs reproducibility, since two runs are not the same conversations.

Adversarial share is a property worth requesting by number. A score on a set that is one third adversarial and a score on a set drawn from typical traffic are different claims about safety, even at the same value on the same scale. The process that produces such items is covered under red-teaming.