Two systems changes that cut the cost of knowledge distillation: taking the teacher out of memory
Multiverse Computing proposes two changes for distillation, the most expensive step in shrinking large language models: caching the teacher's top-K logits once, and a new KL loss that never materialises the full vocabulary matrix.
Research