image-codec

How it is built

Architecture overview

An image codec turns a grid of pixels into a much smaller byte string and back. A lossy codec throws away what the eye will not miss; a lossless codec keeps every bit and wins only by predicting pixels well. Every modern design follows the same pipeline: convert to a colour space where the eye's sensitivities separate, predict or transform each block so most of the energy lands in a few numbers, quantise those numbers (the only lossy step), and entropy-code the result with a model that adapts to the content.

The idea that makes it work is that the encoder is free and the decoder is fixed. The bitstream defines only how to reconstruct; the encoder may search as long as it likes for the cheapest description of a picture, and a better search yields a smaller file without any decoder change. JPEG XL's reference encoder documents exactly this: its effort settings 1 to 10 trade encode time for more coding tools and a more exhaustive search, and at the top of the range it runs Butteraugli iterations to tune adaptive quantisation (source: libjxl, encode effort). Two codecs are in common use as named examples here: JPEG XL, standardised as ISO/IEC 18181, and AVIF, which stores an AV1 intra frame in a HEIF container.

Parts:

• Colour and sample layout — move to a luma/chroma space and drop chroma resolution the eye cannot see — Colour space conversion, chroma-subsampling (no page yet)
• Prediction and transform — remove redundancy inside the image — intra-prediction (no page yet), discrete-cosine-transform (no page yet), SIMD programming
• Quantisation — decide how coarse each coefficient is — Adaptive quantisation
• Entropy coding — model and pack the symbols — Arithmetic coding, Asymmetric numeral systems, Huffman coding, context-modeling (no page yet), Fast hash functions
• Encoder decisions — pick modes and block sizes for the lowest cost at a given quality — rate-distortion-optimization (no page yet), Perceptual quality metrics
• Reconstruction filters and delivery — clean up artefacts and show something early — in-loop-filters (no page yet), progressive-decoding (no page yet)

 pixels ─► colour convert ─► (subsample chroma) ─► split into blocks / groups
                                                          │
                      ┌───────────────────────────────────┤
                      ▼                                   ▼
        predict from neighbours (intra)        transform (DCT, variable size)
        residual = block - prediction          coefficients
                      └───────────────┬───────────────────┘
                                      ▼
                           quantise (per-block step)  ◄── RD search + quality metric
                                      ▼
                    context model ► entropy coder (ANS / arithmetic / Huffman)
                                      ▼
               container: header, groups with offsets, metadata, ICC/colour info
 decode:  parse ─► entropy decode ─► dequantise ─► inverse transform ─► + prediction
          ─► loop / restoration filters ─► colour convert ─► display

How the parts fit together

Colour and layout: the first free win

Cameras and screens deliver RGB, but the eye is far more sensitive to brightness than to colour detail, so codecs work on a luma channel plus two chroma channels. Colour space conversion is a 3x3 matrix plus a transfer curve, and its contract is strict: the decoder must undo exactly what the encoder did, including which primaries and transfer function the numbers mean. JPEG XL makes this explicit. Its codestream always carries a fully defined colour space, and in the lossy case it uses the absolute XYB space, from which the decoder can produce sRGB, Display-P3 or Rec.2100 PQ output directly; lossless coding cannot use XYB and signals its own space (source: libjxl, format overview). chroma-subsampling (no page yet) halves the chroma planes in one or both directions (4:2:0 and 4:2:2 are the common layouts) and must record where the chroma samples sit relative to luma, or colours shift by half a pixel.

Prediction and transform: two ways to remove redundancy

Lossy codecs have two tools and use both. A transform (discrete-cosine-transform, no page yet) turns an N by N block into frequency coefficients, with most of the energy in the first few; it is the heart of JPEG and of JPEG XL's VarDCT mode, which uses variable-sized DCTs (source: libjxl, format overview). Intra prediction (intra-prediction, no page yet) instead predicts a block from its already-decoded neighbours and codes only the difference. The AV1 specification defines a directional intra prediction process with angle deltas, plus smooth and Paeth modes, chroma-from-luma, palette mode and intra block copy (source: AV1 specification); an AVIF image is an AV1 intra frame stored in a HEIF-based file (source: AVIF specification), so these tools are what it has.

The two are complementary, not alternatives: a predictor handles smooth areas and edges cheaply, a transform handles texture, and the encoder chooses per block. JPEG XL's second mode, Modular, goes the other way and uses only integer arithmetic with predictors, reversible colour transforms, palette transforms and a modified Haar transform called Squeeze, which makes lossless coding possible and enables progressive decoding (same source). Inner loops (the transform butterflies, the prediction kernels, the filters) are written with SIMD programming because the same operation repeats over every pixel.

Quantisation: where the loss happens

Quantisation divides each coefficient by a step and rounds. It is the only place information is destroyed, so it is where visual quality is decided. Adaptive quantisation varies the step by block according to local texture: flat areas show blocking and get finer steps, busy areas mask the error and get coarser ones. JPEG XL carries a map of quantisation weights as a Modular sub-bitstream (source: libjxl, format overview) and its encoder at effort 5 and above turns on adaptive quantisation, chroma-from-luma and the Gabor-like filter (source: libjxl, encode effort). The contract between encoder and decoder is simply the step field: the decoder multiplies back and does not care how the encoder chose it.

Entropy coding: modelling beats the coder

After quantisation most coefficients are zero or small, and the cost of the file is decided by how well the entropy coder predicts them. The coder itself is a choice among Huffman coding, Arithmetic coding and Asymmetric numeral systems:

• Huffman coding assigns whole bits per symbol, decodes with a table lookup and is what JPEG used; it wastes bits on highly skewed symbols such as "zero coefficient".
• Arithmetic coding spends fractional bits and pairs naturally with adaptive probabilities. AV1's decoder is organised around a symbol decoder and cumulative distribution functions that are adapted as the frame is decoded (source: AV1 specification, symbol decoder and CDF update processes).
• Asymmetric numeral systems reach arithmetic-coding ratios at close to Huffman speed. JPEG XL's overview names rANS with context modelling and clustering for its entropy stage, and its effort table shows lossless e1 using Huffman, then ANS from e2 onwards (source: libjxl, xl overview and encode effort).

The block that matters more than the choice of coder is context-modeling (no page yet): deciding which probability table to use for each symbol from already-decoded neighbours. JPEG XL's Modular mode selects contexts with a learned decision tree (the "MA tree"), which the encoder builds at effort 4 and above; below that it uses a fixed tree (source: libjxl, encode effort). Fast hash functions appear around this stage in encoders: hashing neighbourhoods or colour values to find repeated patterns, palette entries or LZ77 matches cheaply. That use is a design choice, not a statement about any particular encoder's internals.

Encoder decisions: rate-distortion and the metric

rate-distortion-optimization (no page yet) chooses among modes, block sizes and quantisation by minimising distortion plus lambda times rate. It needs a distortion measure, and the measure is the weakest part. PSNR is easy and tracks the eye poorly; Perceptual quality metrics such as SSIM and perceptually tuned scores are better targets but cost more to compute inside a search loop. JPEG XL's top effort levels run Butteraugli iterations in the quantisation search (source: libjxl, encode effort), accepting minutes of CPU per image for a smaller file. This is the encoder-side freedom described above, and it is why two files at "the same quality setting" from two encoders can differ in size by a lot.

Filters and progressive delivery

in-loop-filters (no page yet) clean block edges and ringing. AV1 includes a Constrained Directional Enhancement Filter and loop restoration with Wiener filtering in the decoder (source: AV1 specification). JPEG XL has a small blur across block boundaries, the "Gaborish" step, paired with an inverse sharpening in the encoder, and an edge-preserving filter with locally adaptable strength (source: libjxl, format overview). The filter strength is signalled in the stream, so the encoder decides how much smoothing the picture needs.

progressive-decoding (no page yet) lets a viewer show a rough picture before all bytes arrive. JPEG XL's VarDCT always has a 1:8 low-frequency image in its first groups, followed by high-frequency passes, and its frames are split into groups with a table of contents so they can be decoded in parallel or partially (source: libjxl, format overview). Groups in VarDCT are 256 by 256 pixels. The AVIF specification defines layered image items as a separate feature (source: AVIF specification, section 2.3).

Data and control flow

Encoding one photograph to JPEG XL (lossy):

1. Read pixels and the declared colour space; convert to XYB.
2. Split into 256 by 256 groups and, inside them, choose a transform size per block.
3. Estimate per-block quantisation from texture; at higher effort, iterate with a perceptual metric.
4. Quantise, build the context model, entropy-code the coefficients with ANS.
5. Write the header, a table of contents with a bitstream offset for every group, then the groups; optionally add Exif or XMP boxes in the container.
6. Store or serve the file.

Decoding it on a phone:

1. Parse the header and table of contents, read the colour and orientation information from the codestream (not from Exif).
2. Decode the low-frequency image first and show a preview.
3. Decode groups in parallel; entropy decode, dequantise, inverse transform.
4. Apply the smoothing filters, then convert to the display space.

Time goes to transform and filtering on the decode side, and to the mode and quantisation search on the encode side.

Migrating existing JPEGs: JPEG XL can recompress a JPEG file losslessly and restore the exact original file, using a reconstruction-data box (jbrd) (source: libjxl, format overview). This works because VarDCT with only 8x8 DCT can represent the original coefficients, so the step is a re-entropy-coding of data already quantised.

Scaling

In the order you hit them:

1. Encode time per image. Levers: lower the effort level, try a faster encoder such as SVT-AV1 for AV1, cache results by content hash.
2. Decode speed on devices. AVIF depends on an AV1 decoder; dav1d states a goal of the highest speed possible on most platforms (source: dav1d README). JPEG XL is designed for efficient software decoding without hardware acceleration (source: JPEG XL overview, jpeg.org).
3. Memory for very large images. Group-based coding and chunked encoding keep memory bounded; libjxl lists the cases where chunked encoding is disabled (source: libjxl, encode effort).
4. Storage and bandwidth. The usual quality targets are tuned per use case; JPEG.org quotes a typical JPEG XL compression ratio of 20:1 to 50:1 (source: jpeg.org). Re-encode once and serve from a CDN.
5. Client support. Serve several formats by content negotiation, or fall back to JPEG.

Failure modes

Real implementations

The alternative: do not write a codec. Choose JPEG for universal reach, a modern format for savings, and decide by decode support in your clients.

Sources

• JPEG, Overview of JPEG XL (ISO/IEC 18181). jpeg.org
• libjxl documentation: format overview, encode effort, overview
• Alliance for Open Media, AV1 Bitstream and Decoding Process Specification. spec
• Alliance for Open Media, AV1 Image File Format (AVIF). spec
• dav1d. README
• ITU-T T.81, Digital compression and coding of continuous-tone still images (JPEG). PDF
• Google, jpegli: repo; libavif: repo