What is quantization?
Quantization stores a model's weights, and sometimes its activations, in fewer bits than the training precision. A 7B parameter model in 16-bit floats needs about 14 GB of memory; the same weights in 4-bit integers fit in under 5 GB, which is what lets a laptop GPU run it at all. The architecture does not change; the numbers inside it are rounded to a coarser grid, and a small calibration step or a smarter grouping recovers most of the quality lost in the rounding.
What quantization does to the weights
A transformer holds most of its parameters as matrices of floating-point numbers, typically 16-bit (fp16 or bf16) after training. Quantization replaces those floats with a smaller integer type plus a scale factor: an int8 weight uses 1 byte instead of 2, and an int4 weight packs two per byte. At inference the runtime multiplies the integer by the scale to recover an approximation of the original weight, then does the matrix multiply the way it would have anyway. The forward pass is the same computation on rounded numbers.
Two effects follow. Memory shrinks in proportion to the bit width, so a checkpoint that needed 14 GB at fp16 fits in about 7 GB at int8 and closer to 4 GB at int4, which is often the difference between a model running on a single consumer GPU and not running at all. Bandwidth drops too: transformer decoding is bound by how fast weights can be streamed from HBM into the compute units, so a smaller checkpoint decodes faster even when the arithmetic itself is done in fp16. The cost is precision. Rounding fp16 down to 4 bits is lossy, and the whole game in a good quantization scheme is choosing what to round and how, so the loss barely shows up in the eval.
Formats you meet in practice: GGUF, GPTQ, AWQ, bitsandbytes
Post-training quantization is the common case: you take a released fp16 checkpoint and produce a quantized version without further training. GPTQ (Frantar et al., 2022) does this by processing the weights layer by layer and using a small calibration dataset to solve for the rounding that minimises the reconstruction error of each layer's output. AWQ (Lin et al., 2023) instead identifies the small fraction of weights that matter most for activations and protects those, then quantizes the rest more aggressively. Both target 4-bit inference and both ship as libraries you point at a Hugging Face checkpoint.
llama.cpp's GGUF format is where most local inference has converged. It packages weights in k-quant variants (Q4_K_M, Q5_K_M, Q8_0, and so on) that trade file size against quality, and the same file runs on CPU, Metal, CUDA and Vulkan without a separate build. On the training side, bitsandbytes exposes 8-bit and 4-bit (NF4) linear layers as drop-in replacements for torch.nn.Linear, which is how QLoRA fine-tunes a 65B model on a single 48 GB GPU: the base weights sit in NF4, and a LoRA adapter trains in a higher precision on top.
Does 4-bit quantization hurt quality?
On standard benchmarks, a well-quantized 4-bit version of a modern chat model usually lands within a point or two of the fp16 baseline, and int8 is close to lossless. The Hugging Face quantization docs and the GPTQ and AWQ papers report this pattern across LLaMA-family models: perplexity rises slightly, task accuracy drops a little on some benchmarks and matches on others. For most product use cases, that gap is smaller than the gap between two prompts.
The place it bites is long-tail behaviour that the benchmarks do not cover. Rare languages, exact numeric reasoning, tool-call argument formatting and code generation at the edges of the model's training distribution can degrade more visibly than the aggregate score suggests. Run your own eval before you swap an fp16 model for a 4-bit one in production, and keep the fp16 checkpoint around as the reference.
When to quantize, and when not to
Reach for quantization when the constraint is memory or bandwidth: fitting a bigger base model onto the GPU you already have, cutting the per-token cost of a self-hosted deployment, or running a 7B or 13B model on a laptop for local development. Int8 is a safe first step for server inference; int4 (GPTQ, AWQ, or a k-quant GGUF around Q4_K_M) is the standard for local and cost-sensitive setups.
Skip it when you are calling a hosted API: the provider has already picked a serving precision, and there is no knob for you to turn. Skip it when your workload is training rather than inference, unless you are specifically doing QLoRA or a similar quantized-base fine-tune. And skip it as a first move when accuracy is regressing: the fix is usually a better prompt, a stronger base model, or better retrieval, not a lower bit width.
The same 8B model at four precisions
# Llama 3 8B Instruct, common precisions and file sizes
precision bits/weight file size fits on
---------------------------------------------------------
fp16 / bf16 16 ~16.1 GB A100 40GB, RTX 4090 24GB tight
Q8_0 (int8) 8 ~8.5 GB RTX 4090 24GB comfortably
Q5_K_M ~5.5 ~5.7 GB RTX 3060 12GB
Q4_K_M ~4.5 ~4.9 GB M2 MacBook 16GB (Metal)
# same prompt, greedy decoding
$ llama-cli -m Meta-Llama-3-8B-Instruct.Q4_K_M.gguf -p "..." -n 128
> [output within a token or two of the fp16 reference on common prompts,
> and drifting on rare-language or exact-arithmetic prompts you should eval]
Bit width is the whole story on disk: fp16 to Q4_K_M shrinks the checkpoint by about 3.3x. Output stays close to the fp16 baseline on common prompts and drifts on the edges you should be evaluating anyway.
Common questions
Quantization: frequently asked
Is quantization the same as compression?
No. Compression, like gzip, is lossless and has to be undone before you use the file. Quantization is a lossy change of representation that the runtime uses directly, and inference happens on the quantized weights. A quantized checkpoint is smaller on disk and smaller in memory at the same time, which is the point. Compression saves storage; quantization saves runtime memory and bandwidth as well.
Should I pick GPTQ, AWQ or GGUF?
For hosted or GPU-only serving through Hugging Face Transformers or vLLM, GPTQ and AWQ are the two mature choices, and AWQ tends to edge ahead on accuracy at 4 bits for recent LLaMA-family models. For local inference across CPU, Metal and CUDA, GGUF via llama.cpp is the format most tools now expect, with Q4_K_M as a reasonable default. If you are training, look at bitsandbytes NF4 for QLoRA rather than either of the post-training formats.
Does quantization make the model faster?
Usually yes for decoding, sometimes no for prefill. Autoregressive decoding on a GPU is memory-bandwidth bound, so smaller weights stream out of HBM faster and tokens per second go up. Prefill and long prompts are more compute bound, and if the kernel dequantizes on the fly the extra arithmetic can eat the win. The exact tradeoff depends on the runtime, the hardware and the batch size, so benchmark before assuming a speedup.
Can I fine-tune a quantized model?
Not directly: the integer weights do not carry gradients. QLoRA is the standard workaround. You keep the base model in 4-bit (NF4) and train a small LoRA adapter in a higher precision on top, so gradient updates flow into the adapter without touching the quantized backbone. The Hugging Face PEFT library packages this behind a few config flags.
Sources
Where these facts come from
- Hugging Face Transformers: Quantization overview: official overview of the quantization backends supported by transformers, with expected memory savings and accuracy notes
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (Frantar et al., 2022): introduces the layer-by-layer 4-bit post-training scheme used by most GPU inference stacks
- AWQ: Activation-aware Weight Quantization (Lin et al., 2023): the other common 4-bit scheme, which protects the small fraction of salient weights identified from activation statistics
- llama.cpp repository: reference implementation of the GGUF format and the k-quant variants (Q4_K_M, Q5_K_M, Q8_0) used for local inference