Why distillation is expensive

Knowledge distillation, training a smaller student model to match the performance of a larger teacher, is a long-established technique in machine learning. With the recent wave of open source large language models it has become a mainstream topic again. The reason is simple: deploying these models is expensive. The Kimi-K3 model has 2.8 trillion parameters and needs roughly 3TB of VRAM just to load.

Compressing large models into smaller ones and recovering their capabilities through distillation has therefore become standard practice. NVIDIA's Nemotron 3 Puzzle 75B and Multiverse Computing's Hypernova 60B are among the high-quality compressed models produced this way. But the distillation step, which decides most of the final quality, is also the most expensive part of the pipeline.

Where the memory goes

The standard setup is online distillation using the Kullback-Leibler divergence loss, which keeps both teacher and student loaded at the same time. At every training step the teacher runs a full forward pass to produce its output distribution, and the student is trained to match it. This is the most expressive setup, since the full teacher distribution is available, but it is also the most memory- and compute-intensive. Two full-vocabulary tensors have to be held per token position, and the teacher is recomputed on every single step even though its behaviour does not change across a training run.

The example the authors give makes the scale concrete: gpt-oss-120b has a vocabulary of 201,088 tokens. At a sequence length of 32K and batch size 4, the teacher-probability tensor alone has shape 4 by 201,088 by 32,768. In bfloat16 that is already about 50GB of VRAM for a single tensor. Add gradients, activations, model weights and optimiser states, and even a single training iteration demands hundreds of GPUs and careful tensor-parallelism strategies.

The two proposed changes

The work Multiverse Computing presented on 10 August 2026 approaches the problem with two systems-level interventions:

  • Offline top-K logit caching: the highest K logits in the teacher's output are computed once and stored. Because the teacher's behaviour does not change across the run, that record can be reused, and the teacher no longer has to sit in memory alongside the student.
  • A fused chunked KL loss: a memory-efficient KL divergence loss that never materialises the vocabulary-size by sequence-length matrix in full. According to the authors this cuts VRAM use far below what the default implementations in libraries such as PyTorch or NVIDIA Megatron-Bridge achieve.

Why it matters

Together the two changes promise to bring training cost down to the point where long-context healing becomes possible on a single GPU. That means the step of recovering a compressed model's lost long-context capability need not require a cluster of hundreds of GPUs.

The real gain here is not only cheapness but freedom to experiment. When the distillation step is a job that runs for hours on a large cluster and costs accordingly, trying different teacher-student pairings or different data mixes is not practical. Once the cost falls far enough, large-scale experimentation becomes practical. These claims still need independent replication, but the direction is clear: work that would push open source model compression beyond the largest labs.