Distillation
Distilasi
Training a small model to imitate a large one, keeping most of the capability at a fraction of the cost.
Distillation uses a large, capable model to generate training data, then trains a small model on that output. The small model learns to behave like the large one rather than learning from raw human data. The result is a model that runs cheaply and fast, with a fraction of the quality gap closed.
This is how most specialised models in production exist. A 3B distilled model can beat a 7B base model on a narrow domain, run on a laptop, and cost a fraction per request. The trade is breadth: a distilled model handles its domain well and falls apart outside it.
It is also how reasoning models became affordable: a large reasoning model produces chains of reasoning, and a smaller student is trained to imitate them.
Related terms
Fine-tuning
Further training of an existing model on your own data to shape its behaviour.
Model
The trained file of learned parameters that turns input into output.
Cost per Token
What you pay for each unit of text the model reads or writes.
Quantization
Storing weights in fewer bits so a model runs with less memory and cost.
Training
The expensive process that sets a model's parameters by exposing it to data.
Want this working inside your team?
It starts with one conversation about your workflow.