An interim report from the Forecasting Research Institute finds that experts significantly underestimated recent AI progress. But there are areas where their forecasts ran too high, not too low.
Model launches arrive weekly, each with its own chart. This guide shows how to build a small but useful evaluation set for your own work, how to grade it and how to read the result.
An average success rate does not show an agent is reliable. This guide covers the consistency gap, the Pass^k metric that measures it, and how to narrow it.
In none of the documented incidents were agents told to escape; they were given a task, and the escape emerged as the shortest path. A practical guide to the three ingredients of a leak, the four patterns that open a channel, isolation levels and what to monitor.
Methods borrowed from psychological testing show that reducing AI safety benchmarks to a single number is misleading.
Vals, which builds evaluations and benchmarks testing AI models on real-world tasks, has closed a $40 million Series A led by a16z at a $400 million valuation.
Artificial Analysis has launched Optima, a platform letting users build custom benchmarks from their own data and workflows, comparing models on cost and time as well as quality.
DeepSeek's leaderboard-topping V4 Flash completed only 53.8% of complex agent tasks in testing. Thirty hard tasks were run across eight different agent harnesses.
Fields Medallist Timothy Gowers says almost all the famous mathematics problems solved by language models so far were solved with counterexamples rather than proofs.
An eval harness surfaced a pattern qualitative review had missed: AI models display their highest confidence precisely when their answers are wrong.
Meta has confirmed that its Muse Spark model breached another company's systems during a security evaluation. The same accident previously happened at OpenAI and Anthropic.
Google Research finds frontier language models encode nearly all facts but struggle to recall many of them. The error comes from access, not absence.
Princeton and the UK AI Security Institute handed AI agents the research questions from two unpublished NeurIPS papers. Both resulting papers were rejected.