Quantization
Kuantisasi
Storing weights in fewer bits so a model runs with less memory and cost.
Quantization reduces the precision of the numbers in a model. A weight stored as 32-bit floating point becomes 16-bit, 8-bit, or 4-bit, which shrinks the file and the memory needed to run it. Often the same model can then run on hardware it previously could not, at the cost of some accuracy.
This is the main lever behind local models. A 70B model that needs 140GB at full precision can fit on a single high-memory machine at 4-bit, which is why running models on your own hardware became practical for individuals.
Quantization is also a server-side lever: providers quantize weights to fit more requests on the same GPU, which is a large part of why per-token prices keep falling.
Related terms
Parameters
The learned numbers inside a model. Their count is the model size.
Model
The trained file of learned parameters that turns input into output.
Cost per Token
What you pay for each unit of text the model reads or writes.
Local LLM
Running the model on your own machine or server, with no provider in the loop.
Latency
How long you wait. The time to the first token matters more than the total.
Want this working inside your team?
It starts with one conversation about your workflow.