DeepSeek's V4 Flash has topped leaderboards since its release and been called a "total monster" by developers. In real-world testing the picture differs: the model completed only 53.8% of a batch of complex agent tasks.

How the test was set up

Composio ran the model through eight different agent harnesses — among them Claude Code, Codex and OpenCode — on 30 deliberately difficult multi-step tasks. The tasks touched live tools such as Gmail, GitHub, Slack and Google Sheets, meaning real services rather than a synthetic exam environment.

Of 240 total runs, 129 passed. The more striking second figure: only six of the 30 workflows completed successfully across every harness tested.

The real finding: the harness gap

That second number is what the story is actually about. The same model completes the same task on some harnesses and fails it on others. The source of failure is therefore not only the model's capability but the software running it.

The harness determines how the model reaches tools, how it handles errors, how it manages context and what it does when a step fails. None of that is written in the model weights; it is written in the code around them.

Why leaderboards mislead

Model leaderboards mostly measure single-step, closed-ended tasks: one question, one answer, one score. Agentic work is the opposite — dozens of steps, a tool call at each, errors, retries.

A model that is perfect in a single step, succeeding 98% of the time at each of thirty steps, still finishes the whole chain only about 55% of the time. Error accumulates. Leaderboards do not measure that accumulation at all.

What to do

The practical conclusion is that choosing a model is not sufficient on its own:

  • Do not trust a leaderboard without testing the model in your own workflow, with your own tools.
  • Treat the harness as a choice too — the same model gives different results in different harnesses.
  • Measure success across the whole chain, not per task.

The second half of the story makes this more urgent: DeepSeek raised its API prices over the same period. Less reliable results are arriving at a higher cost.

What 53.8% means

In reading that figure it is worth remembering the tasks were deliberately made difficult. Success rates on ordinary agent work would come out higher; the test was not built to measure the easy cases.

Even so, the practical meaning of 53.8% is clear: a system running at that rate cannot go into production without human oversight. If one task in two fails, every output has to be checked — and that checking consumes most of the gain automation was supposed to deliver.

The field's real problem

The finding confirms an argument that has been building in agentic AI over the past year: models are advancing, but the reliability of agent systems is not advancing at the same rate. The gap sits in the harness, in tool interfaces and in error handling.

That is dull engineering work — it has none of the announcement value of shipping a model. But Composio's test suggests the next step of progress is located there.