When a language model answers a factual question incorrectly, two very different things may have happened: either it never learned the fact, or it learned it and cannot reach it. Standard accuracy metrics collapse both into a single failure — even though they point to very different limitations and very different fixes.

New work from Google Research measured that distinction and reached a clear conclusion: the problem is mostly lost keys, not empty shelves.

Why the distinction matters

The difference maps directly onto the remedy:

  • An encoding failure calls for scaling model size or expanding data coverage — expensive, slow paths.
  • A recall failure can be addressed through post-training and inference-time methods, helping the model make better use of what it already holds.

In other words, the cause behind an identical-looking error determines the difference between a multimillion-dollar training run and a few weeks of work.

Knowledge profiling

The researchers introduce a behavioural framework they call knowledge profiling. Its core idea is to shift the unit of analysis from the question to the fact: not whether the model answered a particular question correctly, but what the state of that fact is inside the model.

Each fact is placed into one of five profiles: encoding failure, recall failure, direct recall, recall with thinking, and inference without encoding.

The classification rests on three behavioural notions:

  • Encoding: the model can correctly reproduce the fact in a pre-training-like context
  • Recall: it can retrieve the encoded fact without external cues
  • Recognition: it can identify the correct fact when presented among alternatives

Encoding is measured through proposition completion and contextual questioning, which place the model in contexts similar to those where the fact would naturally appear during pre-training — without revealing the answer.

The benchmark: WikiProfile

To support the analysis, the team built WikiProfile: 2,150 Wikipedia-derived facts, each paired with ten questions probing encoding, recall and recognition separately.

Examined through this framework, frontier models such as Gemini3 and GPT-5 show a consistent picture: they encode nearly all the facts, yet struggle to recall many of them.

What it means

The finding shifts the direction of the hallucination debate. A wrong answer does not mean the model lacks the information; more often it has it and cannot reach it. That thinking steps — eliciting intermediate computations — can rescue some facts supports the same reading, and the framework's "recall with thinking" profile names precisely that case.

The difficulty of measuring

Measuring this distinction is harder than it looks, because we cannot directly observe a model "knowing" a fact. All we have is behaviour: what it answers when we ask.

The framework's solution is to probe the same fact in different forms. If the model cannot produce it when asked directly but recognises it among options, the knowledge is there and the route to it is blocked. The same holds if it can complete the sentence in a context resembling pre-training. Put together, these three measurements produce a far more informative picture than a single accuracy percentage.

The method has a limit of its own: a behavioural framework measures how a model looks from the outside, not what happens inside it. Even so, that is exactly the information an intervention decision needs.