model-compression

How it is built

Architecture overview

A model-compression pipeline takes a trained network and produces a smaller or faster one that is still good enough for a stated task, and proves it with a measured comparison against the original. It has a baseline measurement, one or more compression stages, an optional recovery stage that repairs the damage the compression did, and an evaluation gate that decides whether the result may be registered. The stages are quantization (fewer bits per number), pruning (fewer numbers), distillation (a smaller model trained to imitate the large one) and low-rank factorisation (a big matrix replaced by two thin ones). The one idea that makes it work is that a network's weights are far more precise than its function needs, but not uniformly so: a few channels, rows or layers carry most of the sensitivity, so good methods spend their effort finding those and protecting them, using a small calibration set that stands in for real inputs.

Parts:

• Baseline and gate — the reference score on the target task, and the pass/fail rule — Evaluation metrics, Loss functions
• Post-training quantization — lower the bit width of weights (and sometimes activations) without training — Weight quantization, LU and Cholesky factorization, PCA and SVD
• Quantization-aware training — train with the rounding error in the loop — Quantization-aware training (no page yet), Mixed-precision training
• Distillation — train a student on a teacher's outputs — Knowledge distillation, Loss functions
• Pruning and low-rank — remove weights or factor matrices — Network pruning (no page yet), PCA and SVD
• Recovery by adapters — small trainable corrections on top of frozen compressed weights — LoRA adapters (no page yet), Mixed-precision training
• Packing and export — store low-bit and sparse data compactly so the saving is real — Bit packing and PFOR, Weight quantization

 trained model (fp16/bf16) ─► baseline eval (task score, perplexity, latency, memory)
        │
        ├──► calibration set (a few hundred representative inputs)
        │            │
        ▼            ▼
   choose method:  PTQ (GPTQ, AWQ, SmoothQuant) │ prune (magnitude, SparseGPT, Wanda) │ distil (teacher ─► student)
        │                                        │ QAT (fake-quant in the training loop)
        ▼
   compressed candidate ─► eval again ─► within budget? ──no──► recover (adapters, QAT, distil) or milder setting
                                           │ yes
                                           ▼
                        pack (bit-pack, sparse index) ─► export file ─► register with provenance + measured loss

The pipeline is a loop of "compress, measure, adjust", and the gate is part of the design, not an afterthought: a compressed model that is smaller and wrong is a regression with a good-looking file size.

How the parts fit together

Baseline and gate

Everything is judged against a baseline measured the same way on the same data. Evaluation metrics decides what is measured. For language models, perplexity on held-out text is a fast, sensitive early signal; task scores (accuracy on the benchmarks or your own labelled sets) are the truth, and they are noisier and can hide drops that perplexity shows. Also measure agreement with the original (do the two models produce the same output on the same input), since a shifted model can score the same and still behave differently for a user. Latency and memory belong in the same table, because the reason for the exercise is that they improve.

The gate is a rule written before the experiment: for example, no more than x points lost on each task and no regression on a protected slice (a language, a long-context set, a safety set). Averages hide a damaged slice, so the gate checks slices. Loss functions enters twice: as the measure of reconstruction error inside the methods below, and as the objective of distillation.

Post-training quantization

Weight quantization maps each weight to a small integer plus a shared scale (per tensor, per channel, or per group of weights). Rounding to nearest is the baseline and is fine at 8 bits; at 4 bits and below it loses visible accuracy, and the methods that fix this share one structure: choose the rounding using information about the layer's actual inputs, collected on the calibration set.

• GPTQ quantizes a weight matrix one column at a time and, after rounding a column, adjusts the not yet quantized columns to compensate the error, using the inverse of the layer's input second-moment matrix (an approximate second-order method). The paper reports quantizing 175-billion-parameter models in about four GPU hours to 3 or 4 bits per weight with small loss (Frantar et al. 2022). The numerical heart is a Cholesky factorization of the inverse Hessian, which is why LU and Cholesky factorization is a building block here: it makes the compensation updates stable and cheap, and damping is added to the diagonal so the factorization exists.
• AWQ observes that a small fraction of weight channels matter much more, and identifies them by looking at activation magnitudes, not weight magnitudes. Instead of keeping those channels in higher precision (which is hardware-unfriendly), it scales them up before quantizing and compensates in the activations, an equivalent transformation found from offline statistics, with no backpropagation (Lin et al. 2023).
• SmoothQuant targets quantizing activations as well as weights (W8A8). Activations have outlier channels that are hard to quantize; it moves the difficulty into the weights with a per-channel scaling that is mathematically equivalent (Xiao et al. 2022).
• LLM.int8() handles outlier feature dimensions by splitting them into a separate 16-bit multiplication and doing the rest in int8 (Dettmers et al. 2022). It needs no calibration and is implemented in the bitsandbytes library.

Which method: weight-only 4-bit (GPTQ or AWQ) when the deployment is bandwidth-bound (decoding on a GPU or a device); W8A8 (SmoothQuant) when the target has fast int8 matrix units and compute is the limit. Two practical choices dominate the result more than the method: the group size (smaller groups, more scales, more accuracy, slightly bigger file) and the calibration data (it should look like production inputs; a mismatch gives a model that scores well on the calibration domain and poorly elsewhere).

PCA and SVD appears in two places: as an analysis tool (the singular values of a weight matrix show how compressible it is) and as a compression method on its own, low-rank factorisation, which replaces an m-by-n matrix by m-by-r and r-by-n factors when r is small enough to save parameters.

Quantization-aware training

When post-training quantization loses too much (very low bit widths, small models, activations), train with the rounding in the loop: the forward pass uses quantized values ("fake quantization"), and the backward pass treats the rounding as the identity (the straight-through estimator) so gradients still flow (Quantization-aware training, no page yet). The classic integer-only recipe is Jacob et al. 2018. The network adapts its weights to land near the quantization grid. It costs a real training run, so it comes after post-training quantization has been tried; and it needs Mixed-precision training machinery, because master weights stay in higher precision while the forward pass sees low precision.

Distillation

Knowledge distillation trains a small student to match a large teacher. The student sees the teacher's full output distribution (soft targets, softened with a temperature), which carries more information per example than a hard label: the teacher's confusion between similar classes teaches the student about their similarity (Hinton et al. 2015). For language models the loss is usually the divergence between teacher and student token distributions, optionally mixed with the ordinary next-token loss (Loss functions). Distillation changes the architecture, so it is the one method that can reduce depth and width, not only precision; its cost is a training run and teacher inference on the training set. The student is then a normal model that can be quantized afterwards, and the order "distil, then quantize" is the common one. Distilled models trained on a teacher's generated outputs inherit the teacher's errors, so the gate evaluates against ground truth, not only against the teacher.

Pruning and low-rank

Network pruning (no page yet) sets weights to zero. Unstructured pruning removes any weight and gives the highest sparsity for a given loss but little speedup on ordinary hardware; structured patterns (whole heads or channels, or the 2:4 pattern of two non-zeros in each group of four that some GPUs accelerate) are less flexible but run faster. Classic recipes prune, retrain, then quantize and entropy-code: the Deep Compression pipeline (Han et al. 2015). For large language models, one-shot methods avoid retraining: SparseGPT uses the same second-order idea as GPTQ and reports at least 50 % sparsity with minimal accuracy loss on very large models (Frantar and Alistarh 2023); Wanda scores each weight by its magnitude times the norm of its input activation and needs no weight update (Sun et al. 2023). Pruning and quantization compose, and the evaluation gate should be re-run on the combination, since the errors interact.

Recovery by adapters

After compression, a small amount of fine-tuning can repair damage without touching the frozen compressed weights: LoRA adapters (no page yet) add low-rank matrices to selected layers and train only those (Hu et al. 2021). QLoRA keeps the base model in 4-bit and backpropagates through it into the adapters (Dettmers et al. 2023), which is how a very large model is tuned on one GPU. The gradients and adapter weights use Mixed-precision training. In a compression pipeline the adapters play two roles: recovery (fit the quantized model back to the task) and specialisation (one shared compressed base, a small adapter per task or tenant). Adapters can be merged into the weights for a single-file export, or shipped separately if the runtime swaps them.

Packing and export

A 4-bit value stored in an 8-bit byte saves nothing, so the exporter packs. Bit packing and PFOR provides the techniques: two 4-bit values per byte (or other widths across words), laid out so a kernel can unpack with a few shifts and masks and a SIMD load. Per-group scales and zero points are stored beside the packed block. For sparse models, the surviving values need their positions, stored as integer indices or bitmaps; delta coding and PFOR-style packing of those indices keep the index overhead from eating the saving. The contract with the runtime is the layout: the packing order and the group boundaries must match what the inference kernel reads, so the format is versioned and a golden-file test checks that a packed-then-unpacked tensor equals the original quantized one. The on-device-inference system describes the reader of this format.

Data and control flow

One compression run (post-training):

1. Load the baseline model and record its scores, perplexity, latency and memory on the evaluation sets.
2. Draw a calibration set from production-like text (a few hundred samples is a typical order of magnitude; the method's paper states its own choice).
3. Run the method layer by layer: feed the calibration inputs forward, collect input statistics for the layer, quantize or prune its weights, and pass the layer's quantized output on to the next layer.
4. Pack the result and write the export with its settings (bit width, group size, calibration source, method).
5. Re-run the evaluation; apply the gate; if it fails, try a milder setting (more bits, smaller groups, skip sensitive layers) or add a recovery stage.
6. Register the file with its provenance and measured loss. Time goes mostly to the calibration forward passes and the per-layer solves.

One distillation run:

1. Choose teacher, student shape and training data (often the teacher's domain, plus teacher-generated text).
2. Train the student on the teacher's output distributions mixed with the ground-truth loss.
3. Evaluate against ground truth and against the teacher; iterate on size and data.
4. Hand the student to the quantization flow above if the target needs it.

Scaling

1. Calibration cost is bounded by model size and layer solves: more GPU memory lets you hold bigger layers and batch the calibration; for the largest models the layer-by-layer design means only one layer's worth of working memory is needed at a time.
2. Evaluation cost grows with the number of candidate settings; use a fast proxy (perplexity on a small set) to prune the search, and run full task suites only on finalists.
3. Quality at low bit widths: smaller groups, mixed bit widths per layer (sensitive layers get more bits), better calibration data, then adapters or quantization-aware training.
4. Many models: automate the pipeline as a job with a fixed gate; compress each new base model with the same recipe and compare by the same table.
5. Training cost for distillation and recovery is the real expense and scales like any training run, so it is used last.

Failure modes

Real implementations

The alternative: train a smaller model from scratch at the target size, which gives full control and avoids the accuracy loss of squeezing a large one, at the price of a full training run and often a weaker model than a compressed large one of the same footprint. A second alternative is to serve the large model on more hardware and skip compression; that is the right call when the accuracy gate cannot be met at an acceptable size.

Sources

• E. Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, 2022. paper
• J. Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, 2023. paper
• G. Xiao et al., SmoothQuant, ICML 2023. paper
• T. Dettmers et al., LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, 2022. paper; QLoRA, 2023. paper
• B. Jacob et al., Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference, CVPR 2018. paper
• G. Hinton, O. Vinyals, J. Dean, Distilling the Knowledge in a Neural Network, 2015. paper
• S. Han, H. Mao, W. Dally, Deep Compression, ICLR 2016. paper
• E. Frantar, D. Alistarh, SparseGPT, ICML 2023. paper; M. Sun et al., A Simple and Effective Pruning Approach for Large Language Models (Wanda), 2023. paper
• E. J. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, 2021. paper