Artificial Analysis has launched a platform called Optima: users can build custom benchmarks from their own data and their own workflows. Models are compared not only on quality but on cost and time per task.

The problem with general benchmarks

Model leaderboards run on standard exam sets: maths problems, coding questions, general knowledge tests. Those sets share one weakness — they may have nothing to do with your work.

A model scoring highly on a maths exam does not mean it will summarise your contracts correctly. If you work with documents in a language the benchmark never tested, a ranking says even less.

There is a further problem: popular benchmark sets have been published for years and may have leaked into training data. From outside it is impossible to distinguish a model knowing an answer from a model having seen the question before.

Measuring on your own data

Optima's approach targets both problems at once. You build a set from your own documents, your own questions and your own correct answers; models are measured against that set.

Because the set is not public, the leakage problem disappears as well. Memorisation does not help on an exam nobody has seen.

Why cost and time matter

The platform's second contribution is adding cost and time per task to the comparison. In agentic applications that is as decisive as quality.

The reason: token price alone does not show cost. A cheap but inadequate model makes more attempts to finish a task, takes more steps and ends up costing more. An expensive but accurate model can finish in one pass.

The right question is: what does it cost to complete this task? Price per token is only one component of that.

Who it helps

A tool of this kind has direct value for any team running AI in production:

  • Seeing what the difference actually is on your own work before switching models.
  • Measuring whether a version upgrade has degraded quality.
  • Testing whether a cheaper model does the same job.

Building these measurements by hand is possible but laborious; what a platform really offers is not paying that cost over and over.

The changing meaning of benchmarks

The spread of tools like this looks likely to reduce the weight of model leaderboards. General rankings show whether a model exists and roughly which league it plays in; but the decision to choose one will increasingly rest on your own measurement.

That is a healthy direction. In a market where a single ranking is decisive, model builders start optimising to look good on that ranking — and a measure stops being a measure once it becomes a target.

For small teams

The cost of building your own benchmark is usually overstated. A set of thirty or forty questions with known answers gives an adequate signal for most work. What matters is not the size of the set but that it represents your actual work.

Once built, that set becomes a durable asset: it is re-run on every model change and every version upgrade, and returns an answer in minutes.