When a language model answers a factual question incorrectly, two very different things may have happened: either it never learned the fact, or it learned it and cannot reach it. Standard accuracy metrics collapse both into a single failure — even though they point to very different limitations and very different fixes.

New work from Google Research measured that distinction and reached a clear conclusion: the problem is mostly lost keys, not empty shelves.

Why the distinction matters

The difference maps directly onto the remedy:

  • An encoding failure calls for scaling model size or expanding data coverage — expensive, slow paths.
  • A recall failure can be addressed through post-training and inference-time methods, helping the model make better use of what it already holds.

In other words, the cause behind an identical-looking error determines the difference between a multimillion-dollar training run and a few weeks of work.

Knowledge profiling

The researchers introduce a behavioural framework they call knowledge profiling. Its core idea is to shift the unit of analysis from the question to the fact: not whether the model answered a particular question correctly, but what the state of that fact is inside the model.

Each fact is placed into one of five profiles: encoding failure, recall failure, direct recall, recall with thinking, and inference without encoding.

The classification rests on three behavioural notions:

  • Encoding: the model can correctly reproduce the fact in a pre-training-like context
  • Recall: it can retrieve the encoded fact without external cues
  • Recognition: it can identify the correct fact when presented among alternatives

Encoding is measured through proposition completion and contextual questioning, which place the model in contexts similar to those where the fact would naturally appear during pre-training — without revealing the answer.

The benchmark: WikiProfile

To support the analysis, the team built WikiProfile: 2,150 Wikipedia-derived facts, each paired with ten questions probing encoding, recall and recognition separately.

Examined through this framework, frontier models such as Gemini3 and GPT-5 show a consistent picture: they encode nearly all the facts, yet struggle to recall many of them.

What it means

The finding shifts the direction of the hallucination debate. A wrong answer does not mean the model lacks the information; more often it has it and cannot reach it. That thinking steps — eliciting intermediate computations — can rescue some facts supports the same reading, and the framework's "recall with thinking" profile names precisely that case.