What happened

The transformer architecture has been the field's default for eight years and underpins virtually every large language model in production. A MIT Technology Review analysis published on 10 August 2026 argues that assumption is now under pressure: the attention mechanism that made the architecture powerful has become the main obstacle to what models are being asked to do next.

The problem lies in how the mechanism works. Attention compares words in a text against one another to work out which relate to which. That comparison grows disproportionately as the text gets longer. When a model is asked to track an entire code base, a document archive, or the output of an agent session running for weeks, the compute cost gets out of hand.

Three different ways out

The analysis collects startups attacking the problem along independent lines:

  • Subquadratic (Miami): keeps attention but makes it sparse. It says its model decides on the fly, for each piece of text, which words matter and which do not. The company claims its model, SubQ, rivals mainstream models on tasks including search and coding.
  • Manifest AI (San Francisco): replaces attention outright. Its mechanism, called power retention, keeps a rolling summary of the context window; as new information arrives, less relevant information is dropped.
  • Liquid AI (Cambridge, Massachusetts): does not abandon transformers entirely but pairs them with its own liquid neural network technology. The resulting models are far smaller and consume less energy.

How solid are the claims?

Manifest AI argues an existing transformer model can be converted to its architecture with minimal retraining. To demonstrate this it turned the open source coding model StarCoder into a version called PowerCoder, and released a model named Brumby that it says rivals some versions of Alibaba's Qwen family. Liquid AI, meanwhile, builds models for carmakers including Mercedes that run on the small chips inside vehicles; it says its latest models can run on a Raspberry Pi.

The core caveat is plain: most of these claims come from the companies' own announcements and have not been confirmed by independent benchmarks. Subquadratic's claim to have invented the first sparse attention mechanism that rivals mainstream models is among the ones drawing scepticism in the industry. Sparse attention is not a new idea; researchers have produced many variants over the years, and none has matched dense attention at capturing meaning.

Why it matters

Two concrete pressures sit behind this search. The first is cost: total electricity consumed by data centres is projected to double by 2030. The second is a capability ceiling. Reasoning models work by writing notes to themselves in a kind of scratch pad and reading them back, which further increases the volume of data to stay on top of. Goals such as agents staying on task for weeks, or analysing videos hours long, are economically difficult under the current architecture.

In short, the industry is looking for leverage beyond simply making models bigger. If even one of these startups can demonstrate its claims in independent benchmarks, the cost profile of long-context work could shift substantially. If none can, the transformer will remain the field's default for a while longer.