Moonshot AI, the team behind the Chinese assistant Kimi, has published PerceptionBench, a benchmark that isolates and measures the visual perception of multimodal language models. Its claim is pointed: a significant share of the failures recorded as "reasoning errors" actually happen much earlier, while the model is reading the image.

Separating seeing from thinking

Conventional tests bundle perception, knowledge and reasoning into a single task. PerceptionBench pulls them apart, breaking vision into ten atomic sub-skills. Every question can be answered by looking at the image alone — no reasoning, no outside knowledge.

The ten skill domains are:

  • Visual relation, counting, attributes
  • Depth and 3D, localization, comparison
  • Fine-grained recognition, context integration
  • OCR and hallucination

The categories were not defined up front. The authors first collected the errors models actually made on existing tests, traced each one back to the earliest failed step, and built the taxonomy from those traces. That choice rests on a measurement of its own: across the 42 open-source benchmarks they analysed, error profiles overlap very little, meaning each captures a different slice of visual weakness. No single test, and no small group of tests, covered visual perception as a whole.

The questions look trivial

From an internal pool of more than 17,000 verified questions, Moonshot AI is releasing 3,000 tasks. Sixty percent were derived from attributed model errors; the remaining 40 percent were reformulated using augmented images.

On the surface the tasks seem trivial: working out where a symbol sits on a clock face, counting the flowers inside a red box, or deciding which of two pencil cups shows a grey-pink combination and which is solid pink with a cartoon design.

No model breaks 60 percent

Among the 16 frontier models tested, the highest overall accuracy is 59.7 percent, scored by GPT-5.6 Sol. Kimi K3 follows at 58.5 percent, Claude Fable 5 at 57.2 percent, Gemini 3.1 Pro at 56.2 percent and GPT-5.5 at 55.8 percent. The gap widens among open models: Qwen3.5-397B-A17B reaches 47.5 percent and GLM-4.6V only 32.5 percent.

The authors argue the category-level breakdown matters more than the overall ranking — models with nearly identical aggregate scores diverge sharply across individual sub-skills.

Why it matters

The finding confirms what earlier studies suggested: even the best models stumble on basic visual tasks. The practical consequence is that trying to fix a wrong answer with "better reasoning" often means looking in the wrong place. If the failure begins with looking rather than thinking, that is where the fix belongs.