Quantization explained: what Q4, Q5 and Q8 really mean
Quantization is how big AI models fit on ordinary computers. Learn what the labels mean, what you trade away, and how to choose.
If you've browsed models to run locally, you've seen file names like Q4_K_M or Q8_0. These describe quantization, the reason a model that would otherwise need a data center can run on a laptop.
The core idea
A model is mostly a huge list of numbers called weights. During training, each weight is usually stored at high precision, often 16 bits. Quantization stores those numbers with fewer bits.
- 16 bits per weight: full precision (FP16 or BF16).
- 8 bits per weight: about half the size.
- 4 bits per weight: about a quarter of the size.
Fewer bits mean smaller files, less memory and often faster responses. The cost is a small loss of accuracy, because each number is stored less exactly.
Key idea: Quantization is like saving a photo as a compressed JPEG. At sensible settings you barely notice. Push it too far and quality visibly drops.
The math in one line
Approximate weight memory in gigabytes:
parameters (in billions) × bits per weight ÷ 8For an 8-billion-parameter model:
| Format | Approx. bits per weight | Approx. weights size |
|---|---|---|
| FP16 | 16 | 16 GB |
| Q8_0 | 8.5 | 8.5 GB |
| Q6_K | 6.6 | 6.6 GB |
| Q5_K_M | 5.7 | 5.7 GB |
| Q4_K_M | 4.85 | 4.9 GB |
| Q3_K_M | 3.9 | 3.9 GB |
Bits-per-weight values are approximate averages for common GGUF formats. Real files vary slightly by model.
Reading the labels
In the popular GGUF format (used by llama.cpp and tools built on it, such as Ollama):
- The number (Q4, Q5, Q8) is the approximate bits per weight.
- K refers to the "k-quant" family of methods, which spend more bits on the weights that matter most.
- S, M, L (small, medium, large) are variants within a level. Larger keeps a bit more quality and takes a bit more space.
- Q8_0 is a simpler method that is very close to full quality.
Memory isn't only weights
When a model runs, it also needs memory for:
- The context (KV cache): working memory for the conversation. It grows with context length, so a 32,000-token context needs much more than a 4,000-token one.
- Runtime overhead: buffers and the program itself.
That's why a 4.9 GB model file may need 6 GB or more to run comfortably. Our Local Model Memory Estimator includes rough allowances for both.
How to choose
- Start with Q4_K_M. It's a widely used balance of size and quality.
- Have memory to spare? Try Q5_K_M or Q6_K for a quality bump.
- Need the best output and have lots of memory? Q8_0 is near full quality.
- Tight on memory? A larger model at Q4 often beats a smaller model at Q8, but test both on your own tasks.
- Avoid going below Q3 unless you have no other option; quality tends to drop noticeably.
Tip: The best test is your own work. Run the same few prompts on two versions and compare the answers side by side.
Summary
Quantization trades a little accuracy for a lot of efficiency. Understanding the labels lets you pick the best model your hardware can actually run, instead of guessing.