Quantization: FP32, FP16, INT8, and 4-Bit Models

Introduction

A 7-billion-parameter model needs about 28 GB of memory if you store each weight as a 32-bit float. Your laptop has 16 GB. Nothing is wrong with the model — only with how the numbers are stored.

Quantization is the fix: represent weights with fewer bits per number. It is the reason a 7B model runs on a gaming GPU, why inference costs dropped an order of magnitude, and why “4-bit” appears in every local-LLM tutorial. This mission gives you the memory math, the trade-offs, and the engineering judgment to choose precision deliberately.

Learning objectives

• Explain what FP32, FP16, BF16, INT8, and 4-bit formats actually store.
• Compute the memory footprint of a model at any precision.
• Describe how quantization preserves quality (scaling, calibration, grouping).
• Choose precision per scenario: training, serving, local inference, edge.

The formats: what the bits buy

Floating-point numbers split their bits between exponent (range) and mantissa (precision). Fewer total bits means cutting both — the trade-off is range versus detail.

Format	Bits/weight	Exponent : Mantissa	Typical use
FP32	32	8 : 23	Training (classic), reference quality
FP16	16	5 : 10	GPU training/inference (risk of overflow)
BF16	16	8 : 7	GPU training/inference (FP32 range, less detail)
INT8	8	integer + scale	Serving quantization (CPU & GPU)
4-bit (NF4/GPTQ/AWQ)	4	packed codes + scales	Local LLM inference, memory-bound serving

The analogy that makes it stick: measuring with rulers. FP32 is a laser meter — precise to a fraction of a millimeter. FP16 is a tape measure — good to a millimeter. INT8 is a carpenter’s ruler marked in millimeters only. 4-bit is a ruler marked every centimeter. For most questions (“how long is this board?”) the cheap ruler gives the same answer; for a few questions, only the laser will do. Quantization is the art of knowing which questions are which — and making the cheap ruler slightly smarter so it is right almost always.

The memory math

The core formula is simple — parameters × bits ÷ 8 = bytes:

$$ \text{GB} = \frac{\text{parameters} \times \text{bits}}{8 \times 1024^3} $$

Model	FP32	FP16/BF16	INT8	4-bit
1B params	4.0 GB	2.0 GB	1.0 GB	~0.5 GB
7B params	28 GB	14 GB	7 GB	~3.5 GB
13B params	52 GB	26 GB	13 GB	~6.5 GB
70B params	280 GB	140 GB	70 GB	~35 GB

Two engineering corrections to the naive math:

1. Weights are not the whole story. Inference also needs memory for activations and, critically, the KV cache — which grows with context length and batch size. A 7B model at 4-bit fits in 4 GB of weights, but a long conversation can still exhaust an 8 GB card. Budget for context, not just parameters.
2. Scales and metadata add overhead. Real 4-bit formats store group scales alongside packed weights, so the true footprint is a few percent above the ideal — plan for ~3.8–4.2 GB for a 7B model, not exactly 3.5.

How quantization preserves quality

If you just rounded every weight to fewer bits, small weights would collapse to zero and the model would degrade badly. Modern methods add structure:

• Scaling. Map the weight range onto the integer range with a per-tensor (or finer) scale factor: w_quantized = round(w / scale). Dequantization multiplies back. The outlier problem — one huge weight wrecking the scale for everyone — is handled by finer granularity.
• Granularity. Scales per tensor → per channel → per group (e.g., every 64 or 128 weights). Finer groups isolate outliers; this is the main quality lever in 4-bit.
• Calibration. Run representative data through the model and observe actual weight and activation ranges, then choose scales that minimize real error (GPTQ-style).
• Information-theoretic formats. NF4 (used by QLoRA) assumes weights follow a normal distribution and allocates code points where weights actually live — quantile quantization, not uniform rounding.
• Mixed precision. Keep the most sensitive layers (first/last, embeddings, norms) at higher precision; quantize the bulk. Quality-critical exceptions are cheap.

The result, empirically: INT8 is usually indistinguishable from FP16 on benchmarks; well-executed 4-bit costs roughly 1–3% quality on general tasks — a trade most applications happily make for 4× memory savings. Poorly executed 4-bit (wrong group size, no calibration) is noticeably worse, which is why “which 4-bit recipe” matters.

The trade-off table

Dimension	FP16	INT8	4-bit
Memory (7B)	14 GB	7 GB	~3.5 GB
Relative speed	baseline	often faster (memory-bound)	often faster
Quality loss	none	~0%	~1–3% (done well)
Hardware support	universal	broad	GPU kernels (bitsandbytes, llama.cpp)
Typical setting	cloud serving	high-throughput serving	local / memory-bound

The counterintuitive part: LLM inference is usually memory-bandwidth-bound, not compute-bound — the GPU spends its time streaming weights from memory. Fewer bits means fewer bytes to stream, so quantized models are frequently faster, not just smaller. This is why production serving often runs INT8 or FP8 even where memory is plentiful.

Choosing precision: the decision guide

• Training / fine-tuning — BF16 (or FP32 master weights). Never quantize what you are updating. (Exception: QLoRA — Level 8 — quantizes the frozen base to 4-bit and trains small adapters in BF16.)
• Cloud API serving — you usually do not choose; providers serve optimized precision and price accordingly.
• Self-hosted serving, memory available — FP16/BF16 for maximum fidelity, INT8 for throughput.
• Self-hosted, memory constrained — 4-bit (GPTQ/AWQ/NF4). A 70B model at 4-bit fits in 40 GB — two consumer-class GPUs instead of a server.
• Local development / laptop — 4-bit via llama.cpp, Ollama, or LM Studio. The quality delta is irrelevant for prototyping prompts and pipelines.

The senior-engineer framing: quantization is a deployment decision driven by a quality budget. Measure your task’s evaluation set (Level 7) at each precision and pick the cheapest one that passes. “4-bit is fine” is a measurement, not a vibe.

Hands-on exercise

Feel the compression yourself — no GPU required:

1. In Python/NumPy, create a random array of 1M floats from a normal distribution (pretend these are weights).
2. Implement naive INT8 quantization: scale = max(abs(w)) / 127, quantize, dequantize. Compute the mean absolute error and the compression ratio (exactly 4×).
3. Now implement grouped scaling: split into groups of 64, each with its own scale. Compare the error — this is the granularity effect, visible in one line of code.
4. Simulate an outlier: set 0.1% of weights to 100× the typical magnitude. Compare per-tensor vs. grouped error. You have just discovered why NF4 and GPTQ exist.
5. Bonus: quantize to 4 bits (16 levels) with grouping and plot error vs. group size.

import numpy as np

def quantize_int8(w: np.ndarray) -> tuple[np.ndarray, float]:
    scale = np.abs(w).max() / 127.0
    q = np.clip(np.round(w / scale), -127, 127).astype(np.int8)
    return q, scale

def dequantize(q: np.ndarray, scale: float) -> np.ndarray:
    return q.astype(np.float32) * scale

Key takeaways

• Quantization stores weights with fewer bits: FP32 → FP16/BF16 → INT8 → 4-bit, trading precision for memory and bandwidth.
• Memory math: parameters × bits ÷ 8 — a 7B model is ~14 GB at FP16, ~3.5 GB at 4-bit. Always add KV cache and activation memory to the budget.
• Quality is preserved through scaling, granularity, calibration, and mixed precision — not naive rounding. INT8 ≈ lossless; good 4-bit costs ~1–3%.
• Inference is memory-bandwidth-bound, so quantized models are often faster too.
• Treat precision as a deployment decision validated against an evaluation set — the cheapest format that passes your quality bar wins.

Continue your journey

Next mission: Local Models, GPU Memory, and Model Serving (coming soon) — or continue to LLM APIs: Prompts, Messages, and Model Parameters (coming soon). Full journey: roadmap overview.