A perfect score means nothing if the test-taker knew the questions in advance. That is how Google DeepMind describes the core problem in evaluating AI models: benchmark contamination. If a model already saw the test questions during training, you can only trust its result so far.

Against that, the company says it has launched the first double-blind evaluation of a proprietary frontier AI model. External tests stay locked inside a cryptographic "box," so the model cannot later take those questions and optimise itself specifically for that test.

The trade-off it targets

According to DeepMind, highly sensitive external evaluations used to require a compromise. There were two options and each hurt someone:

  • The evaluator hands over its test prompts. The model provider then sees the questions in advance and the door to contamination opens.
  • The provider hands over its model weights. It then puts its intellectual property at risk.

There is a recent example of the dilemma: the ARC-AGI benchmark evaluation of Anthropic's Fable 5 was delayed, because the company enforces a 30-day data retention policy for its strongest models.

How it works

The double-blind evaluation is meant to remove that trade-off. Google uses Confidential Space, part of its own cloud's confidential computing portfolio, to do it. The setup cryptographically verifies that both the external test data and the model stay private to their respective owners: the evaluator never sees the Gemini weights, and Google never sees the test prompts.

The pilot runs a model from the Gemini Flash Lite line against confidential benchmarks. The company lays out the methodology and results in a technical report.

Until now, the confidentiality of external prompts rested on zero-logging protocols and contractual safeguards. Adding technical and cryptographic protection on top, the company says, is a significant step for secure model evaluation.

The method has a limit too: the double-blind setup prevents the test leaking afterwards, but it does not guarantee that nothing resembling those questions sits in the model's training data. Part of the contamination happens not at exam time but months earlier, while the web is being scraped.

Why it matters most in sensitive areas

DeepMind says the method pays off most in highly sensitive evaluations: cybersecurity testing, or audits run by government agencies. Independent organisations could rigorously test advanced models without giving up data sovereignty or security.

The deeper point is this: model evaluation is steadily turning into a regulatory function. A test measuring a model's offensive cyber capability is no longer an academic curiosity; it becomes an input to a government's decision on whether to permit that model at all. Leaking such a test's questions does not merely void the test — it voids the decision.

What changes

Google says it hopes the effort sets a new standard for model oversight and helps the industry build more reliable, widely trusted systems. A company proposing as a standard a method that eases its own auditing naturally invites a question.

But the technical claim is checkable: the cryptographic setup either genuinely closes both sides off or it does not, and that can be examined independently. The value of the method will depend less on who proposed it than on who can implement it.