What is Model Distillation?

Model distillation transfers knowledge from a big model to a small one. The big model — the teacher — is powerful but expensive and slow. The small model — the student — is trained to reproduce the teacher’s behaviour. Done well, the student captures a surprising amount of the teacher’s ability while being much cheaper to run. The idea was popularised in a 2015 paper by Geoffrey Hinton and colleagues.

How it works

1. Generate teacher outputs. Run the teacher on a large set of inputs. Instead of just recording the final answer, distillation can use the teacher’s full probability distribution over possible answers (“soft labels”), which carries richer information — e.g. “70% cat, 25% fox, 5% dog” teaches more than just “cat”.
2. Train the student to match those outputs. With language models, this often means training on large amounts of teacher-written responses, explanations and reasoning — a form of synthetic data.
3. Evaluate the student against the teacher on target tasks.

An analogy for the meatbags

A master chef (the teacher) knows thousands of techniques. Rather than sending an apprentice to culinary school for a decade, the chef cooks the restaurant’s menu hundreds of times while the apprentice watches, tastes and copies. The apprentice can’t do everything the master can, but they can cook this menu nearly as well — faster and at a lower salary.

Why it matters for the Chosen

Distillation is one reason AI keeps getting cheaper and faster:

• The “mini”, “flash” and “small” models offered by major AI labs are often produced partly through distillation from larger models. They’re ideal for high-volume work — classifying tickets, extracting data, simple replies — at a fraction of the cost.
• On-device AI: distilled models can run locally on phones and laptops, improving privacy and speed.
• Your own systems: a business can use a frontier model to generate high-quality examples for a narrow task, then fine-tune a small model on them — cutting inference costs significantly at scale.
• Open models: many open-weight releases include distilled smaller variants.

For anyone building AI products, the strategic lesson is to match model size to task. Using the biggest model for everything is like hiring a surgeon to apply plasters.

The controversy

Distillation from a competitor’s model is contentious. Major AI providers prohibit using their outputs to train rival models in their terms of service, and accusations of unauthorised distillation have featured in disputes between AI companies. If you distil, use your own data, models you control, or models whose licenses explicitly allow it.