By Gustavo TorresIn June 2023, the public leaderboard of open models at Hugging Face, the largest platform for sharing language models, gave LLaMA 65B a score of 48.8% on MMLU (Massive Multitask Language Understanding), a multiple-choice exam of 14,042 questions across 57 subjects. LLaMA 65B was the large model Meta had released that February, and the number is its size: 65 billion parameters. Meta's own paper reported a good deal more, and the gap was far too big to be chance. Hugging Face looked into why and found three programs that all claimed to grade the same exam. Given the same model and the same questions, MMLU's original code scored it at 63.6%, Stanford's HELM benchmark (Holistic Evaluation of Language Models) at 63.7%, and the LM Evaluation Harness ("LM" for language model), the evaluation tool from the open research collective EleutherAI that the leaderboard relied on, at 48.8%.
It wasn't only the score that changed; the ranking did too. Under the Harness, Falcon 40B, a model from the Technology Innovation Institute in Abu Dhabi, finished ahead of LLaMA 65B, 52.7% to 48.8%. Under the original code it came third, behind LLaMA 30B as well. And the weakest models, which barely cleared the 25% you'd get by answering at random under the original code, gained almost ten points under the Harness.
Eight models, the same exam and three programs grading it, as reported by Hugging Face in June 2023. Every model's score changes, and so does who comes first.
A multiple-choice exam looks like the one case where evaluating a language model is a solved problem. Normally the hard part is deciding whether an answer is right, because there's rarely a single correct one, and that takes other judges: a program that checks, a person, another model. In a multiple-choice exam, every question has a key, a single letter, and grading is a matter of comparing letters. That's why benchmarks with fixed answers (fixed collections of questions used to compare models) are the most common way to compare models, and why their figures lead the model card, the page that describes a model, of every new release.
Even so, the score depends on four decisions that aren't published alongside it: how the answer is read, how the question is asked, which questions are used (and whether the model had already seen them) and how many there are. Any one of them can move the result by more than the gap between the models being compared. This article takes them one at a time, using real MMLU questions and two small models measured on a laptop, and ends with what to ask before trusting the number on a model card. It's the second in a series on how language models are evaluated. The first introduces the different kinds of judge, but you don't need to have read it.
What's inside a benchmark
A fixed-answer benchmark has four parts. The questions. The answer key, which holds the correct answer to each one. A scoring rule, which says how the model's answer is extracted and when it counts as matching the key. And an averaging rule, which turns thousands of hits and misses into a single number. The first two are the exam, and they get published. The last two are the measuring instrument, and they almost never appear next to the score.
MMLU is the textbook example. A group led by Dan Hendrycks at the University of California, Berkeley published it in 2020: 14,042 test questions with four options each, spread across 57 subjects from abstract algebra to virology, with between 100 and 1,534 questions per subject. Even the average comes in two versions: over all 14,042 questions, where professional law, with 1,534, weighs fifteen times as much as abstract algebra, or over the 57 subjects, where each counts equally. Each subject also comes with five solved questions that aren't scored and serve as examples. This is how the model sees a question, with the five examples in front of it:
The following are multiple choice questions (with answers) about high school geography.
The rate of natural increase of a population is found by subtracting the
A. crude death rate from the crude birth date.
B. crude birth rate from the crude death rate.
C. doubling time from the crude birth rate.
D. fertility rate from the crude death rate.
Answer: A
[… four more solved questions in the same format …]
What family do most European languages belong to?
A. Ural-Altaic
B. Basque
C. Indo-European
D. Phoenician
Answer:The five solved examples don't teach the subject; they teach the format. This is in-context
learning: the model sees the pattern of question,
options, Answer: and letter five times over, and carries it on. That's why MMLU is said to be
evaluated with five examples (5-shot); without them it would be 0-shot.
The text stops at Answer:, and that's as far as the exam goes in fixing anything. The model hasn't
answered yet, because a language model doesn't answer; it continues text. For the next position it
assigns a probability to every piece of its vocabulary, the tokens (words, word fragments or
single symbols). Reading the answer means deciding what to do with that list of probabilities, and
there's more than one way to go about it.
To see this with numbers anyone can reproduce, I measured two small open models: SmolLM2-360M, from Hugging Face, with 360 million parameters and a vocabulary of 49,152 tokens, and Qwen2.5-0.5B, from the Chinese company Alibaba, with 500 million parameters and a vocabulary of 151,665. They're base models, not tuned for chat, which is the kind of model MMLU was designed for. The measurements use 570 MMLU questions, ten picked at random from each subject with a fixed seed, always the same ones and always with the same five examples. The program downloads the models and the questions, takes every measurement in the article and saves every probability, so anyone can repeat or check them.
Three ways to read the answer
The three programs Hugging Face compared didn't differ in the exam but in what they did with the
model's probabilities after Answer:. With the same model, prompt and question, the three readings
can give three different answers, as the figure shows with the geography question from the example
and SmolLM2.
One question, one model and three ways of reading its answer. With the options shown, SmolLM2 spreads its probability almost evenly over the four letters; without them, it gives "Indo-European" 99.6%.
The most likely letter
MMLU's original code looks at only four of those probabilities, the ones for the letters A, B, C and D, and takes the highest. The model never writes anything: a single pass through the model per question is enough, and the result is always one of the four options.
Both models put almost all of the next token's probability on those four letters, 99.4% on average
for SmolLM2 and 99.8% for Qwen2.5: the five examples have taught them that a letter follows
Answer:. But saying a letter doesn't mean knowing which one. On the geography question, SmolLM2
gives the four letters between 23% and 27% each, and across the whole exam it scores 24.7%, no
better than answering at random: it picks A on 245 of the 570 questions and B on 216, whatever the
question. Qwen2.5, by contrast, scores 52.3%, its best result under any of the readings in this
article, and on the geography question it gives C 92%.
Even this reading has a trap. A token includes the space in front of it, and in Answer: A the
token after the colon isn't "A" but " A", with the space. (The lesson on
BPE, byte pair encoding, the algorithm that builds models'
vocabularies, explains how text is split into tokens.) A program that compares the probabilities of
"A", "B", "C" and "D" without the space is looking at tokens the text never puts there, and the score
changes. Qwen2.5 drops from the 52.3% it gets when the letter is read properly, with the space, to
48.9%: 19 fewer questions out of 570, a small difference but bigger than its margin. SmolLM2 goes
from 24.7% to 27.2%, which is noise. That the mistake costs so little is exactly what makes it
dangerous: the figure looks reasonable, and nobody checks it.
The most likely whole option
The 2023 Harness did something different: it computed the probability of each complete option as a continuation of the text, " C. Indo-European" in full, and took the most likely one. The probability of a sequence is the product of its tokens' probabilities, each below 1, so a long option starts at a disadvantage simply for being long. That's why the result is usually divided by the number of tokens, or of characters. Even so, that probability also reflects how ordinary the phrase is, not only whether it's the right answer.
There's a variant that doesn't show the options. The prompt is only the question and Answer:, and
each option is scored as a possible continuation, like a fill-in-the-blank exercise. It's called the
cloze formulation, after that kind of exercise, and it's the classic way to evaluate a model that
can't answer with a letter.
With the options shown, SmolLM2 stays at chance: 24.0%, or 24.9% when dividing by tokens. Without them, and dividing by characters, it climbs to 38.2%, and on the example question it gives "Indo-European" 99.6% of the probability among the four options. The model knows more than the letter reading lets on; what it can't do is tie each answer to its letter. Qwen2.5 goes the opposite way: from 52.3% when the letter is read, it drops to 47.2% with the whole option, 39.6% dividing by tokens and 35.4% without the options.
What the model writes
HELM let the model generate text and compared it with the correct letter. If the most likely token isn't a letter but a line break or a "The", the answer counts as wrong even when the correct letter had more probability than the other three. With base models and five examples in front of them this hardly ever happens: for both models the most likely token was a letter on all 570 questions, so this reading agrees exactly with the most likely letter.
Even so, it's the most widely used reading today, for two reasons. Many models can only be used through an API (application programming interface, the interface one program uses to call another program's service) that doesn't return probabilities, or returns them for only a few tokens. And chat and reasoning models don't answer with a letter: they explain, reason and finally give an answer that has to be found in the text, usually by requiring a fixed format such as "Answer: B" and searching for it with a pattern. An answer in any other format counts as wrong, so this reading also measures whether the model sticks to the format.
Which reading is fair
Both models, read in five ways on the same 570 questions. Reading the letter, Qwen2.5 beats SmolLM2 by 27.5 points; without the options, the two are indistinguishable.
The two models respond in opposite ways. Qwen2.5 does better the more the reading looks like a multiple-choice exam: 52.3% when the letter is read and 35.4% without the options, almost 17 points less. SmolLM2 beats chance only when it doesn't see the options. Reading the letter, Qwen2.5 is 27.5 points ahead on the same questions; without the options, SmolLM2 even comes out ahead, by 2.8, a difference that fits inside the ±3.4 margin. Depending on how the answer is read, Qwen2.5 is either far better than SmolLM2 or indistinguishable from it. That's what happened to LLaMA 65B and Falcon 40B in Hugging Face's table, on a small scale and with two models anyone can run.
Joshua Robinson and colleagues put a name to this in 2022. Reading the letter only works if the model can tie each option to its symbol, a skill they call multiple choice symbol binding, and it varies a great deal from one model to the next. Models that have it do far better when the letter is read, because they can take in all four options at once and compare them. Models that lack it only show a signal when the options are hidden. OLMES (Open Language Model Evaluation Standard), the evaluation standard that the Allen Institute for AI (AI2), a non-profit research institute in Seattle, proposed in 2024, resolves the dilemma by evaluating every model both ways and keeping the better score. The differences are large. Llama 3 70B, Meta's 2024 model, gets 93.7% on ARC-Challenge (AI2 Reasoning Challenge), the institute's own benchmark of school science questions, when the letter is read, and 69.0% when the options are hidden. And OLMo 7B (Open Language Model), one of the institute's open models, answered with letters at random for the first part of its training; the ability only appeared after about 400 billion tokens.
That explains the Hugging Face table from the start of the article. The 2023 Harness read the whole option, which gives some signal to models that can't tie answers to letters, and that's why the weakest ones rose from 25% to over 33%. LLaMA 65B, which can, was instead judged on how likely each full phrase was, and lost fifteen points. No reading is right in the abstract. Each measures a slightly different ability, and the minimum requirement is to use the same one for every model being compared.
How the question is asked
Reading the answer is half the instrument. The other half is the text put in front of the model, which moves the score even when nothing about what's being asked changes.
The simplest example is the order of the options. The same four options can be presented in any order, and the correct answer shouldn't change. To check, I ran the 570 questions four more times, rotating the options so that the correct answer always sat at A, then always at B, and so on to D, and read the most likely letter.
The same 570 questions, with the correct answer moved to each of the four letters. SmolLM2 does well only when the key lands on the letter it was going to say anyway; Qwen2.5 varies by 8.6 points.
SmolLM2 scores 44.2% when the correct answer is at A and 8.4% when it's at C. That isn't because it knows more when the answer is A: it says A or B on four questions out of five, and it's right whenever the key happens to fall where it was going to answer anyway. Qwen2.5, which does tie letters to options, depends far less on the order. Its score ranges from 46.0% to 54.6% depending on where the key sits, with a slight preference for C, which it picks in 29.6% of its answers. Even so, the 8.6 points between its best letter and its worst exceed its margin of error.
In 2023, Chujie Zheng and colleagues measured this bias in 20 models. With every correct answer placed at A, LLaMA 30B scored 68.2% on MMLU; with every one at D, 41.2%. That's a 27-point gap on the same questions. By their analysis, the main cause is that the model starts out giving certain letters more probability, whatever the options say. The fix is cheap: shuffle the options several times and average the results, or estimate that preference and subtract it.
Formatting matters even more. Melanie Sclar and colleagues (2023) changed details that leave a prompt's meaning untouched, such as the separator between question and answer, the spacing or the capitalisation, and the accuracy of LLaMA-2-13B, a Meta model, on a single task swung by as much as 76 points from one format to another. A study by Moran Mizrahi and colleagues (2024) tested 20 models on 39 tasks, using hundreds of different wordings of the same instructions, and the ranking of the models shifted with the wording: on one task, the same model came first with one wording and ninth with another. That's why a standard like OLMES fixes each task's exact prompt and examples, and why scores are best reported across several prompts or, at the very least, with the same prompt for every model.
If the model has already seen the exam
Benchmarks are public and live on the web, and models are trained on the web. MMLU's questions are in the paper's repository, and also in copies, translations and forum threads where someone asks about one of them. If the test questions, or their answers, ended up in the training data, part of the score measures memory. That's contamination.
Whoever has the training data can search it for the benchmark. In 2020, the authors of GPT-3, OpenAI's language model of that year, looked for runs of thirteen consecutive words that appeared in both the benchmarks and the data they were about to train on, and removed the matches. A bug in their code let some through, and as they explained in their paper, retraining the model was too expensive, so they also scored each benchmark with the suspect questions left out. Searching for exact runs has another limit. A 2023 study on rephrased questions trained a 13-billion-parameter model on rewritten or translated versions of the questions from MMLU and other exams, and the model matched GPT-4 without the match search detecting anything.
Whoever doesn't have the data, which is nearly everyone, can only question the model, and there are two ideas for doing so. The first looks at order. A benchmark's questions are published in a particular order, but any other order would be equally valid, so a model that hasn't seen them has no reason to prefer one. A model that memorised them does: it assigns more probability to the published order than to shuffled ones. The test by Oren and colleagues (2023) compares the two probabilities. It works on models from 1.4 billion parameters up, with around a thousand examples, and applied to five public models it found little evidence of widespread contamination. The second idea looks at unlikely words. Text a model hasn't seen tends to contain a few words it considers very improbable, while text it saw in training hardly has any. Min-K% Prob (2023) scores a text by the average probability of its K% least likely tokens. Neither is conclusive for a single question, but across hundreds they give a signal.
The most direct check is to write new questions. GSM8K (Grade School Math 8K) is a widely used benchmark of about 8,500 primary-school maths problems. In 2024 the data company Scale AI wrote GSM1k, 1,205 new problems matching its style and difficulty, and kept them unpublished. Some model families lost up to 8 points going from one to the other.
Three practices help prevent contamination. The first is to mark the files. BIG-bench (Beyond the Imitation Game), a collaborative 2022 benchmark with more than two hundred tasks, puts a canary string in every file, a unique identifier that anyone preparing training data can search for in order to exclude those files. The second is to keep the test questions unpublished and grade on a server. The third is to use questions newer than the models' training data: LiveBench (2024) is refreshed every month with problems from recent maths competitions, papers and news, and checks all of its answers without judges. An evaluation of your own, built from questions that have never been published, is safe by construction, until the day someone pushes it to a public repository.
How many questions you need
A score over questions is an estimate: a sample-based measurement of the share of correct answers the model would get on all questions of this kind. If the model gets a share right, the 95% margin of error is roughly
With the 570 questions in these measurements, every score carries a margin of about ±4 points: ±3.5 at 25% and ±4.0 at 40%. The 13.5 points SmolLM2 gains when it can't see the options are a real difference; the 0.7 separating the letter reading from the whole-option reading are not. Comparing the two models on the same questions, Qwen2.5's 27.5-point lead over SmolLM2 when the letter is read has a margin of ±5.5, and SmolLM2's 2.8-point lead without the options has a margin of ±3.4.
Comparing on the same questions
What usually matters isn't a single score but the difference between two, and there the margin depends on how the comparison is made. A small example shows how. Albarán, as we'll call a company that makes invoicing software, has an assistant that answers questions from its support team, and it tests two new versions on 200 real, closed questions. Each answer is a yes or no plus a date, and a program checks them against the key. Version 1 gets 70% right (140 questions) and version 2 gets 74% (148).
If the two versions had each been tested on a different set of 200 questions, the margin on the difference would be about ±8.8 points, and a 4-point lead would mean nothing. But they answered the same questions, so what counts is how many of them the two disagree on. For each question, write down if only version 2 gets it right, if only version 1 does, and if both get it right or both get it wrong. The lead is the average of those numbers, and its margin comes from the same idea as before, applied to those 200 numbers. Questions on which the two agree contribute zeros and add no noise: each question's difficulty affects both versions equally and cancels out.
The margin of error on a 4-point lead over the same 200 questions, by the number of questions on which the two versions disagree. If version 2 changes only a few answers, the lead is real; if it changes many, it's noise.
Suppose version 2 fixes 10 answers and breaks 2. The two versions then disagree on 12 questions and the margin is ±3.4 points, so the lead is real. If it fixes 14 and breaks 6, the lead is still 8 questions, but now they disagree on 20, the margin rises to ±4.4, and you can no longer tell. Up to 16 disagreements, the 4 points exceed their margin. If the two versions failed independently, without sharing which questions they find hard, they'd disagree on about 81 and the margin would climb back to ±8.8. Evan Miller works this out, along with the rest of this section, in Adding Error Bars to Evals (2024), and recommends comparing on the same questions whenever you can.
Questions that come in groups
The formula assumes every question is independent of the others, and that isn't always the case. In a reading-comprehension benchmark, several questions share the same passage, and a model that understands the passage tends to get all of them right, while one that doesn't tends to get all of them wrong. There's less information in the results than it seems, and the margin has to be calculated per group. Miller finds that the grouped margin can be more than three times the naive one: 3.05 times on DROP (Discrete Reasoning Over Paragraphs), a benchmark of questions about Wikipedia paragraphs.
Answers that change from one run to the next
If the model generates its answer by sampling among the most likely tokens, the same question can get different answers in two runs, and the score moves for that reason too. There are two remedies: answer each question several times and average the results or, when you have access to them, use the probabilities instead of sampling, which removes that part of the noise. What Miller advises against is lowering the sampling temperature just to make the score move less, because then you're measuring a different model, one nobody uses that way.
How many questions
All of this can be turned around for planning: decide what difference you want to be able to detect and work out how many questions it takes. Under the assumptions of a typical example of Miller's, detecting a 3-point difference between two models with reasonable probability (80%) takes about a thousand questions, which is why he recommends that a new benchmark have at least that many.
When the exam runs out of road
A score has a floor and a ceiling, and both sit further from 0 and 100 than they seem.
The floor is chance. With four options, answering at random gets you 25%, and always answering D gets you 26.9% on MMLU, because D is the key for 26.9% of its questions. A score of 30% doesn't mean "30% knowledge"; it's barely more than nothing. That's why the 2024 version of Hugging Face's leaderboard, when it switched exams, rescales the scores so that chance is worth 0 and a perfect score is worth 100. On a four-option exam, a raw 60% becomes 46.7.
The ceiling is the exam's own mistakes. A 2024 review, Are We Done with MMLU?, estimated that 6.5% of MMLU's questions contain an error (a wrong key, an ambiguous question, more than one correct option), rising to 57% in virology. Even the solved examples aren't spared: the first one in the geography prompt above says "crude birth date" where it should say "crude birth rate". A perfect model wouldn't score 100%, and near the ceiling, part of what separates two models is which one agrees more with the key's mistakes. In June 2024, Anthropic and OpenAI published the same 88.7% for their flagship models, Claude 3.5 Sonnet and GPT-4o.
When a benchmark saturates, a harder one takes its place. MMLU-Pro (2024) goes from four options to ten, which brings chance down to 10%, and replaces recall questions with ones that require reasoning. Models lose between 16 and 33 points going from MMLU to MMLU-Pro. As a side benefit, the score depends less on the prompt: across 24 different prompt styles, the variation drops from 4 or 5 points to 2. Hugging Face's leaderboard made the switch in 2024, replacing its six exams, MMLU among them, with six harder ones, MMLU-Pro among them. Its explanation: the best models were close to human level, some had probably been trained on the questions, and some exams contained errors.
How to read the number on a model card
In December 2023, Google launched Gemini Ultra, its most capable model, with 90.0% on MMLU, above the 89.8% estimated for a human expert. The table in the technical report set it beside GPT-4's 86.4%, but the two figures came from different procedures. GPT-4's 86.4% was the one OpenAI had published, with five examples and a single answer. Gemini's 90.0% used what the report calls CoT@32: the model reasons before answering (chain of thought), does so 32 times and, if enough of the answers agree, takes the majority answer; if not, it answers directly without reasoning. In its appendix, the same report compares the two models under one procedure, measuring GPT-4 through OpenAI's API. With a single answer, picking the most likely token at every step, Gemini Ultra scored 84.0% and GPT-4 84.2%. With the 32-answer procedure applied to both, they scored 90.0% and 87.3%. Same exam, same two models, and which one wins depends on how the question is asked.
Nothing in the table is false, and the procedure is spelled out in it. But the figure that travels is the headline one, so with any benchmark score the thing to do is ask how it was obtained. The questions are always the same.
- Which program, and which version? The 2023 LM Evaluation Harness and today's don't give the same results.
- How was the answer read? The most likely letter, the whole option or the generated text.
- How many examples, which prompt, and with or without reasoning?
- One answer or several? And if several, how they're combined.
- On which questions? The whole exam or part of it, and how the results were averaged.
- With what margin? And if it's compared with another model, was it on the same questions?
- Could the model have seen the exam? If nobody checked, nobody knows.
Two figures are only comparable if they come from the same program with the same configuration. The figures on a model card were measured by whoever built the model, with their own program and configuration; those on another card were measured by another team with theirs.
Takeaways
If you remember one thing, make it this: a benchmark score is a measurement, and a measurement depends on the instrument. The exam fixes the questions and the key, but not how the question is asked, how the answer is read, which questions count or how many there are, and any of those four decisions can move the score by more than the gap between two models.
| before trusting the score, ask… | why |
|---|---|
| how the answer was read | LLaMA 65B scores 63.6% or 48.8% on MMLU; reading the letter, Qwen2.5 beats SmolLM2 by 27.5 points, and without the options the two are indistinguishable |
| which prompt, how many examples and in what order | moving the correct answer from A to D shifts LLaMA 30B's score by 27 points; a change of format can shift it by up to 76 |
| whether the model could have seen the questions | some families lose up to 8 points on new problems (GSM1k) |
| on how many questions, and whether the comparison used the same ones | on 200 questions, a 4-point gap can be noise or a real difference |
| where the floor and the ceiling are | with four options, chance gives 25%; 6.5% of MMLU's questions have errors |
And if you remember more, make it these practical rules.
Fix the instrument before comparing. Use the same program, version, prompt, examples and way of reading the answer for every model, and write them down next to the score.
Read the answer in a way the model is able to give. A model that can't tie answers to letters scores no better than chance if you read the letter, and one that can loses points if you read the whole phrase. If you're comparing very different models, measure both ways, as OLMES does, and say which one you used.
Shuffle the options. If the score changes when you move the correct answer around, you're measuring the model's taste for a letter.
Compare on the same questions and put a margin on the difference. What matters is how many questions the two models disagree on.
Be wary of an exam everyone aces. Near the ceiling, the score measures the key's mistakes; near the floor, little more than chance.