camera-isp

How it is built

Architecture overview

A camera image signal processor (ISP) turns the raw output of an image sensor into a picture a person or a model can use. The sensor delivers one number per pixel behind a colour-filter mosaic (Bayer), noisy and uneven across the frame. The ISP subtracts the black level, repairs bad pixels, flattens lens shading, interpolates the missing colours, maps sensor colour to a display colour space, compresses brightness into the output range and sharpens, then hands YUV or RGB frames to an encoder, a display or a neural network. The idea that makes it a system and not a filter chain is the loop around it: the ISP writes statistics about every frame, a control process reads them and picks the next exposure, gain, white balance and focus, and those settings reach the sensor and ISP a few frames later. In libcamera the pixel work is hardware and the Image Processing Algorithms run on the CPU, implement 3A (auto exposure, white balance, focus) and "close the control loop of the ISP" (source: libcamera architecture docs).

Parts:

• Sensor and link — pixel array, binning, exposure lines, analogue gain, blanking, raw Bayer over MIPI CSI-2 — sensor driver (not a block yet)
• Front end — crop, early downscale, statistics while pixels stream in — line-buffer-streaming (no page yet), dma-transfers (no page yet)
• Bayer pipe — black level, defect pixels, lens shading, white-balance gains, denoise, HDR stitch — image-denoising (no page yet), SIMD programming, Deterministic fixed-point math
• Demosaic — one colour sample per pixel becomes three — demosaicing (no page yet)
• RGB pipe — colour matrix, tone curve, gamma, sharpening, output colour space, scaling — Color space conversion, Tone mapping, convolution-image-filters (no page yet), image-warping-remap (no page yet)
• 3A control — metering, white-balance estimation, focus search, turned into sensor and ISP settings once per frame — auto-exposure-focus-white-balance (no page yet)
• Multi-frame processing — burst merge, temporal denoise, stabilisation — image-alignment-registration (no page yet), video-stabilization (no page yet), image-pyramids (no page yet)
• Neural stage — learned denoise inside the pipeline, hand-off to detection — Convolutional networks, Weight quantization

 scene ─► lens ─► ┌────────┐ raw Bayer ┌───────────┐ raw tiles ┌───────────────┐  YUV / RGB
                  │ sensor ├─(CSI-2)──►│ front end ├─── DRAM ─►│   back end    ├──► encoder
                  └───▲────┘           │ crop+stats│           │ Bayer pipe    │    display
   exposure, gain,    │                └─────┬─────┘           │ demosaic      │    detector
   blanking           │                      │ statistics      │ RGB pipe      │
                      │                      ▼ buffer          └───────▲───────┘
                      │    ┌─────────────────────────────────────┐     │ parameter
                      │    │ 3A control loop (CPU, once a frame) ├─────┘ buffer
                      └────┤ AE · AWB · AF · shading · tuning    │
                           └─────────────────────────────────────┘

The pixel path is hard real time (sensor lines arrive at a fixed rate and cannot be paused); the control loop is soft real time with a latency of whole frames. Every design choice below keeps the second from disturbing the first.

How the parts fit together

Sensor and link

libcamera's sensor model, based on the MIPI CCS specification, has a pixel array, an analogue crop, optional binning (combining same-colour neighbours) or skipping, a digital crop and the frame as sent on the bus (source: libcamera camera sensor model). The sensor is the first actuator of the loop: a raw sensor driver must expose V4L2 controls for analogue gain, exposure (in image lines), blanking and pixel rate, and a per-sensor helper turns the gain code into a real gain (source: libcamera sensor driver requirements). Blanking and pixel rate set the frame duration, hence the longest exposure at a given frame rate. Contract with the next stage: a known Bayer order, black level and bit depth; a flip can change the order, so drivers should report it with V4L2_CTRL_FLAG_MODIFY_LAYOUT (same source).

Front end and statistics

In Raspberry Pi's PiSP the front end receives pixels from the camera, does minimal processing, writes to memory, can crop and rescale into two outputs and gathers statistics for the 3A algorithms. It runs in hard real time with good quality of service to SDRAM (source: PiSP specification). That is why it is separate from the back end: the part that must keep up with the sensor stays small, and heavy work runs later from memory where stalls are allowed.

The control loop sees only the statistics. The Raspberry Pi guide describes a matrix of weighted luminance regions plus a histogram, on Pi 5 a 16x16 grid with integer weights 0 to 15 (source: Raspberry Pi Camera Algorithm and Tuning Guide). Android's model likewise assumes statistics come from raw sensor data (source: Android camera HAL docs).

Stages exchange frames or tiles in DRAM by DMA (DMA and IOMMU) as dmabuf file descriptors, which libcamera lists among its helpers, so frames pass on without copying where hardware allows. dma-transfers and line-buffer-streaming (both no page yet) are the graph's blocks for this seam.

Bayer pipe

PiSP's Bayer pipe is a good map of what belongs before demosaic: defective pixel correction, green equalisation (the two greens of the mosaic behave differently), temporal and spatial denoise, black level, lens shading, white-balance gains, colour denoise, chromatic aberration correction, de-binning, HDR stitch with tone map, then demosaic (source: PiSP specification). Reasons to stay in the Bayer domain:

• Noise has not yet been spread across three channels by interpolation and the colour matrix. image-denoising (no page yet) covers bilateral, non-local-means and temporal methods.
• Black level and shading belong to the sensor and lens, not the scene. Uncorrected shading darkens the edges; luminance-only correction leaves a cyan cast growing outward; and full luminance correction "can boost noise more than we might like", so Raspberry Pi's ALSC always corrects colour shading fully and luminance only in part (source: Raspberry Pi tuning guide).
• A software path runs the same kernels with SIMD programming. Where a hardware and a software path must give the same pixels, Deterministic fixed-point math applies: explicit formats, defined rounding.

PiSP's back end works on tiles (hardware maximum width typically 640 pixels) with overlapping source rectangles because filters need context. This trades line memory against re-reading the overlap and lets the back end interleave work, for example 4K60 video with a best-effort JPEG still (source: PiSP specification).

Demosaic

demosaicing (no page yet) reconstructs the two missing colours per pixel. Bilinear interpolation is cheap; a fixed linear filter with gradient correction needs only a small window of lines, which suits streaming. Malvar, He and Cutler's linear filters improve PSNR by over 5.5 dB against bilinear and outperform most nonlinear algorithms, without the artifacts of nonlinear processing and at much lower cost (source: Malvar et al., ICASSP 2004, abstract). Alternatives are edge-adaptive nonlinear methods and learned joint demosaic-denoise networks. After this stage: three channels per pixel.

RGB pipe

PiSP applies a colour correction matrix, a saturation control that "prevents hue distortion as saturation of different colour channels occurs", conversion to YCbCr, false-colour reduction and sharpening, the inverse conversion, gamma, an output colour space, downscale (factor 1, or 2 to 8) and a resampler (source: PiSP specification).

• Matrix and gamma. The matrix maps sensor to output primaries in linear light, so gamma comes after it (Color space conversion).
• Tone. Global tone mapping is a curve; local tone mapping needs a blur or pyramid. libcamera's feature matrix lists local tone mapping as supported on Raspberry Pi 5, in development on i.MX8MP, RK3399 and Renesas, and absent in the Pi 4 and IPU3 hardware (source: libcamera docs). See Tone mapping and image-pyramids (no page yet).
• Detail. Sharpening and false-colour suppression work on luma and chroma separately (convolution-image-filters, no page yet).
• Geometry. Image resampling filters explains why the kernel width must scale with the shrink factor. Undistortion and stabilisation are remaps (image-warping-remap, no page yet).

3A control

auto-exposure-focus-white-balance (no page yet) decides the numbers the other stages use. In the Raspberry Pi guide:

• Exposure. A weighted average of regional luminance is driven to a target, with histogram constraints on top; the algorithm controls shutter time, analogue gain and digital gain together.
• White balance. A maximum-likelihood choice among feasible illuminants measured from a grey target. The guide notes that illuminants lie near the Planckian locus and that fluorescent and LED light fit that less well.
• Pacing. Each algorithm has a Prepare step before the ISP runs and a Process step when statistics arrive. Both should take "considerably less than a millisecond"; heavy work (AWB, ALSC) runs on an asynchronous thread, and Prepare filters outputs toward targets, for example with an IIR filter (IIR and FIR filters).

The sensor changes settings only at frame boundaries, so the controller acts once per frame at a known phase (Fixed-rate control loop). libcamera's exposure design note shows the cost of latency: with a two-frame delay in its worked example, copying the "current" auto value into a manual request flickers unless the controller waits until the mode change has been applied (source: libcamera AE design document).

Frame orchestration

On Linux a pipeline handler drives the Media Controller and V4L2 devices; IPA modules (the 3A algorithms) are plug-ins that can run in a sandboxed process and see image data only through dmabufs the handler passes (source: libcamera architecture docs). The kernel side is concrete in the Rockchip ISP1 driver: capture nodes, a metadata capture node with 3A and histogram statistics, and a metadata output node that takes per-frame parameters in a fixed or an extensible format (source: kernel rkisp1 docs). The Request API ties a whole pipeline configuration to specific buffers, so frames with different settings can be queued ahead; without it user space would have to drain the pipeline between frames (source: kernel media docs). Android defines the same contract as requests that carry all capture settings and 3A modes, processed in order, each yielding metadata and buffers (source: Android camera HAL docs).

Multi-frame and neural stages

Google's HDR+ captures a burst of constant-exposure frames and aligns and merges them on raw Bayer data rather than ISP output (to "circumvent the ISP's unwanted tone mapping and spatial denoising"), with FFT-based alignment and a hybrid 2D/3D Wiener filter; it takes 4 seconds for 12 megapixels on the device (source: Hasinoff et al. 2016). So the ISP must hand raw frames and metadata to software; image-alignment-registration (no page yet) and image-denoising run there, with the transform from Fast Fourier transform. Video has a cheaper relative: PiSP's temporal denoise combines the current frame with a stored long-term-average frame (source: PiSP specification). video-stabilization (no page yet) adds motion estimation, path smoothing and a warp.

A small Convolutional network can replace denoise or demosaic; libcamera allows neural networks in in-tree modules only if model and training data are open (source: libcamera architecture docs). At the output, the Edge AI camera system consumes resized frames for a detector, usually with Weight quantization.

Data and control flow

One frame in a video stream:

1. The sensor integrates for the exposure set several frames earlier and reads out line by line over CSI-2.
2. The front end crops, optionally scales, and writes raw pixels and a statistics buffer. Time is bounded by the sensor line rate; a stall loses lines.
3. The frame-end event reaches the pipeline handler, and the statistics go to the IPA. On Raspberry Pi 4 and earlier the statistics come out of the ISP run, so Process runs after the frame is produced; on Pi 5 the front end collects them, so Process runs before Prepare programs the back end (source: Raspberry Pi tuning guide).
4. Process digests the statistics; Prepare computes ISP parameters and, if needed, sensor controls for a later frame.
5. The back end reads the raw tiles, runs Bayer pipe, demosaic and RGB pipe with the parameter buffer, and writes up to two outputs.
6. Buffers go to the consumer by file descriptor; result metadata (exposure, gain, colour temperature) returns to the application.

Where time goes: at 30 fps the period is 33.3 ms (arithmetic). For scale, libcamera's software ISP documentation shows an example log of 8143 us per frame and reports about 7.8 ms per frame for 1920x1080 SGRBG10 data on one laptop with the CPU fixed at 2 GHz, counting statistics and debayering only (source: libcamera software ISP benchmarking). Hardware ISPs run at pixel rate and spend time as latency, not CPU.

A scene change (lights come on):

1. Frame N is over-exposed; its histogram shows clipping, and AE computes a new target, filtered toward, not jumped to.
2. The exposure reaches the sensor and takes effect a few frames later (a property of sensor and driver).
3. AWB sees the new illuminant and moves gains gradually; the colour matrix follows the colour temperature. The result is a fade over several frames; tuning trades speed against flicker.

A still with a burst: the viewfinder runs with 3A; the shutter yields several raw frames at one low exposure; they are aligned (FFT-based in HDR+) and merged in the raw domain with a hybrid 2D/3D Wiener filter; the merged raw then goes through demosaic, matrix and tone mapping and on to a JPEG encoder (the Image codec and Video encoder systems).

Scaling

1. Memory bandwidth. Every inter-stage pass is a DRAM write and read. Arithmetic: a 3840x2160 frame at 10 bits per pixel is about 10.4 MB raw, about 311 MB/s per write at 30 fps, again for each full-frame pass. Levers: tiles or line buffers instead of frames, early downscale, and compression between stages (PiSP's front and back ends share a compressed raw format to reduce memory bandwidth, source: PiSP specification).
2. Pixel rate at the sensor. Binning and skipping cut the rate before the ISP sees it (source: libcamera sensor model); pick the sensor mode for the output actually needed.
3. Compute for software ISPs. Vectorise first (SIMD programming); libcamera has demonstrated a GPU-accelerated software ISP (source: libcamera news, ELCE 2025). Power is a budget too: libcamera's benchmark guide measures watts next to frame time.
4. Control latency. Faster convergence needs fewer frames of delay: per-frame requests, a shorter pipeline.
5. Tuning effort. Every sensor and lens needs its own black levels, shading tables, matrices and AWB curves (Raspberry Pi: a tuning tool and a JSON file per camera). This grows with the number of products, not with frame size.
6. Several cameras. One front end serves one streaming camera, so two cameras need two front ends; one back end can multiplex their frames at sub-frame level (source: PiSP specification).

Failure modes

Real implementations

Sources

• libcamera docs: architecture, IPA guide, exposure design, sensor model, feature matrix, software ISP benchmarking, sensor drivers, news.
• Raspberry Pi: Camera Algorithm and Tuning Guide, PiSP specification.
• Linux kernel: Rockchip ISP1, Request API, Media Controller; Android Camera HAL, requests and pipeline; MIPI CSI-2 (named only).
• H. Malvar, L. He, R. Cutler, High-quality linear interpolation for demosaicing of Bayer-patterned color images, ICASSP 2004. DOI, abstract
• S. W. Hasinoff et al., Burst photography for high dynamic range and low-light imaging on mobile cameras, ACM TOG 35(6), 2016. DOI, abstract