Quantization
Compressing a model by reducing the numerical precision of its weights, cutting memory and speed cost.
Quantization converts model weights from 16-bit or 32-bit floats down to 8-bit, 4-bit, or even 2-bit integers. A 4-bit quantized 70B model fits in 40GB of memory instead of 140GB — the difference between needing an H100 cluster and running on a single consumer GPU.
Modern quantization (GPTQ, AWQ, GGUF, FP8) preserves 95–99% of original accuracy on most benchmarks. The quality cost is real but usually acceptable, especially for inference-time deployments.
For enterprises, quantization is the primary lever that makes self-hosted open-source LLMs economically competitive with proprietary APIs.
Key points
- 4-bit quantization is the current production sweet spot
- Preserves 95–99% of accuracy on most tasks
- FP8 on H100/H200 gives speed and memory gains with minimal quality loss
- Some tasks (long-form generation, coding) are more sensitive than others
Common use cases
Frequently asked
Related terms
Applying Quantization in your business?
We help enterprises design, ship, and govern AI systems end-to-end.