Weight quantization

Deep dive

How it works

In: a weight matrix W (fp16), group size G, bit width b, optional calibration activations · Out: packed b-bit integers q plus a scale (and zero point) per group · State: the packed weights and scales; the original W can be dropped · Cost: quantize O(P) for round-to-nearest, GPTQ hours for 70B+; inference reads bytes ÷ (16/b)

Asymmetric group-wise quantization to b bits (q ∈ [0, 2^b − 1]) for each group of G consecutive weights along the input dimension:

scale = (max − min) / (2^b − 1)
zero  = round(−min / scale)                 clamp to [0, 2^b − 1]
q     = clamp(round(w / scale) + zero, 0, 2^b − 1)
ŵ     = (q − zero) · scale

dot(ŵ_group, x_group) = scale · (Σ q·x − zero · Σ x)    one multiply per group, not per weight

Smarter rounding with the same storage format:

• GPTQ: quantize the columns of W one at a time. After each one, spread its rounding error onto the not-yet-quantized columns using the inverse Hessian H = 2XXᵀ of the layer inputs (Optimal Brain Surgeon). 175B models quantize in about 4 GPU hours.
• AWQ: about 1% of input channels (those with large activations) dominate the error. Scale those channels up before quantizing (and the activations down), with a per-channel scale found by grid search. No backprop, no reconstruction.
• SmoothQuant (W8A8): migrate activation outliers into the weights with per-channel scales, so both quantize to int8 for integer GEMMs.
• LLM.int8(): int8 matmul, with the few outlier feature dimensions (magnitude > 6) computed in fp16.

Code

Group-wise int4 with packed nibbles and a fused dequant dot product: the core of a weight-only GEMV kernel.

const G: usize = 128;                      // group size along the input dimension

pub struct Q4 { packed: Vec<u8>, scale: Vec<f32>, zero: Vec<u8> }   // 2 weights per byte

pub fn quantize(w: &[f32]) -> Q4 {
    assert!(w.len() % G == 0);
    let mut q = Q4 { packed: vec![0; w.len() / 2], scale: vec![], zero: vec![] };
    for (gi, g) in w.chunks(G).enumerate() {
        let (lo, hi) = g.iter().fold((f32::MAX, f32::MIN), |(a, b), &x| (a.min(x), b.max(x)));
        let (lo, hi) = (lo.min(0.0), hi.max(0.0));            // keep 0 exactly representable
        let s = if hi > lo { (hi - lo) / 15.0 } else { 1.0 };
        let z = (-lo / s).round().clamp(0.0, 15.0);
        q.scale.push(s);
        q.zero.push(z as u8);
        for (j, &x) in g.iter().enumerate() {
            let v = ((x / s).round() + z).clamp(0.0, 15.0) as u8;
            let idx = gi * G + j;
            q.packed[idx / 2] |= v << ((idx & 1) * 4);        // low nibble = even index
        }
    }
    q
}

/// dot(dequant(q), x) without materializing fp weights: scale·(Σq·x − zero·Σx) per group.
pub fn dot(q: &Q4, x: &[f32]) -> f32 {
    let mut acc = 0.0;
    for (gi, xg) in x.chunks(G).enumerate() {
        let bytes = &q.packed[gi * G / 2..(gi + 1) * G / 2];
        let (mut qx, mut sx) = (0.0f32, 0.0f32);
        for (b, xs) in bytes.iter().zip(xg.chunks(2)) {
            qx += (b & 0x0F) as f32 * xs[0] + (b >> 4) as f32 * xs[1];
            sx += xs[0] + xs[1];
        }
        acc += q.scale[gi] * (qx - q.zero[gi] as f32 * sx);
    }
    acc
}

fn main() {
    let mut seed = 0x9E3779B97F4A7C15u64;
    let mut rnd = || { seed ^= seed << 13; seed ^= seed >> 7; seed ^= seed << 17;
                       (seed >> 11) as f32 / (1u64 << 53) as f32 * 2.0 - 1.0 };
    let n = 4096;
    let w: Vec<f32> = (0..n).map(|_| 0.02 * rnd()).collect();
    let x: Vec<f32> = (0..n).map(|_| rnd()).collect();
    let q = quantize(&w);
    let exact: f32 = w.iter().zip(&x).map(|(a, b)| a * b).sum();
    let bits = (q.packed.len() * 8 + q.scale.len() * 16 + q.zero.len() * 4) as f32 / n as f32; // fp16 scales on disk
    let abs_err = (dot(&q, &x) - exact).abs();
    let scale: f32 = w.iter().zip(&x).map(|(a, b)| (a * b).abs()).sum();
    println!("bits/weight {bits:.3}, |err| {abs_err:.2e} vs Σ|w·x| {scale:.2e}");
    assert!(abs_err < 0.02 * scale);
}

A GPU kernel does the same per warp: load 16 B of packed weights (32 int4), unpack with shifts and masks, or use lop3 tricks that build fp16 values directly from nibbles, then FMA against x in registers. Marlin-style kernels pre-shuffle the weight layout offline so that each load maps straight onto tensor-core fragments.

Complexity and constants

• Decode tokens/s ≈ HBM bandwidth / (weight bytes + KV bytes read per step). 7B: fp16 14 GB gives about 240 tok/s upper bound on H100; int4 about 3.9 GB gives about 850. Real kernels reach 60-85% of that.
• Format overheads: int4 g128 with fp16 scale + 4-bit zero ≈ 4.16 bpw; Q4_0 4.5; Q8_0 8.5; Q4_K (super-blocks of 256 with 6-bit sub-scales) about 4.5.
• Quantization time: round-to-nearest takes seconds; AWQ minutes; GPTQ hours for 70B+ on one GPU.

Efficiency

• Cost: batch-1 decode reads every weight once, so tokens/s ≈ HBM bandwidth / bytes. 70B in fp16 is 140 GB, about 42 ms per token at 3.35 TB/s; in 4 bits about 37 GB, about 3.5x faster. For 7B: about 240 tok/s upper bound in fp16 against about 850 in int4 on an H100, real kernels reaching 60-85% (order of). Bound: memory bandwidth; dequantization hides under it.
• Levers:
  • 4-bit groups of 128 with GPTQ or AWQ — weight bytes about ÷ 3.5, about 0.1-0.3 perplexity (above) — memory-bound decode
  • 8-bit embedding, head and edge layers — small share of bytes, large share of the error — every quantized model
  • fp8 KV cache — halves the other memory stream — long contexts
  • W8A8 (SmoothQuant) or fp8 at batch above about 16-32 — native tensor-core GEMMs instead of dequant-in-GEMM — compute-bound serving
  • fast packed kernels (Marlin, GGUF K-quants) — most of the bandwidth bound — GPU or CPU serving
• Naive → standard → state of the art:
  • fp16 weights · 140 GB, about 42 ms per token for 70B
  • round-to-nearest int8 or int4 groups · seconds to quantize, 2-4x fewer bytes
  • GPTQ or AWQ int4 with Marlin-class kernels · about 3.5x faster decode (above)

Implementation notes

• Quantize along the input (K) dimension, so a group's scale is constant within the dot-product accumulation. Keep scales in fp16/bf16.
• Leave the embedding, LM head and sometimes the first and last layers at 8 bits or higher: small share of bytes, large share of the error.
• Calibration for GPTQ/AWQ: about 128 sequences of 2048 tokens from a distribution close to deployment. Calibrating on wiki text and serving code hurts.
• At batch > about 16-32, dequant-in-GEMM becomes compute-heavy. Switch to fp8 W8A8 (Hopper) or int8 SmoothQuant kernels, which use native tensor cores.
• The KV cache is the other big memory stream at long context. fp8 KV halves it with small loss.

Pitfalls

• Symmetric quantization of skewed groups wastes half the range. Use a zero point, or clip to percentiles.
• Outlier channels (magnitude ≈ 100x the median) blow up the group scale. Smaller groups, AWQ scaling or rotations (QuaRot/SpinQuant-style Hadamard transforms) fix it.
• Perplexity can look fine while long-context, math or code accuracy drops. Evaluate on the target tasks.
• Nibble order and packing layout differ between formats (GPTQ, AWQ, GGUF, Marlin). Converting without repacking gives garbage that still "runs".
• Mixing a quantized base with LoRA adapters trained on the fp16 base: merge before quantizing, or train the adapter on the quantized base (QLoRA).

Variants and alternatives

In production

llama.cpp/GGUF (Q4_K_M, Q8_0 …), vLLM (GPTQ/AWQ via Marlin kernels, fp8), TensorRT-LLM (int4 AWQ, fp8, NVFP4), bitsandbytes (LLM.int8(), NF4), Hugging Face Transformers, ExLlamaV2, MLX on Apple silicon.

Implementations

Sources

• Frantar, Ashkboos, Hoefler, Alistarh, GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, ICLR 2023. paper · code
• Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, MLSys 2024. paper · code
• Dettmers, Lewis, Belkada, Zettlemoyer, LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, NeurIPS 2022. paper · code
• Xiao et al., SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, ICML 2023. paper · code
• Dettmers, Pagnoni, Holtzman, Zettlemoyer, QLoRA: Efficient Finetuning of Quantized LLMs, NeurIPS 2023. paper · code