Mean win rate

updated August 17, 2026

Mean win rate scores a model by how often it beats the other models in the evaluated cohort, averaged across tasks. It is a relative ranking statistic, so a model's score changes when the cohort changes even if its underlying performance does not.

The computation is pairwise. On each task, every model is compared against every other model in the run; the model's win rate on that task is the fraction of those comparisons it wins; the reported score is the mean of those per-task win rates. The appeal is aggregation: tasks scored with accuracy, rubric credit, and exact match sit on incompatible scales, and win rate converts all of them into a common unit before averaging.

MedHELM (Stanford CRFM) is the healthcare suite built on this metric. It covers 121 clinical tasks across 5 categories and 22 subcategories, drawn from 31 datasets organized in a clinician-validated taxonomy, and ranks models by mean win rate. Version 5.0.0 was last updated May 14, 2026, and the Stanford-led maintainers refresh it on a roughly quarterly cadence.

The metric's cost is that no score is absolute. Adding a stronger model to the cohort lowers every incumbent's mean win rate without any model changing. Removing a weak model does the same. Two mean win rates published at different refreshes are therefore not comparable, and a mean win rate cannot answer whether a model clears a threshold, only which model ranked higher against the cohort present at the time.

Procurement criteria that need a fixed bar should be written against an absolute metric such as rubric credit or task success rate, with mean win rate used for shortlisting. See composite index for the related problem of weighted aggregation across benchmarks.