realtime-renderer

How it is built

Architecture overview

A real-time renderer turns a scene (meshes, materials, lights, a camera) into one image every 16 ms or less, and does it again for the next frame. Almost all of its design is about not doing work: skip objects the camera cannot see, draw coarser geometry far away, light each pixel with only the lights that reach it, and let the GPU decide all of this so the CPU does not become the bottleneck. The renderer is built from four layers. A visibility stage reduces the scene to what is on screen. A geometry stage feeds the GPU meshes at the right level of detail. A frame graph schedules the passes (depth, shadows, lighting, post effects) and their resources. A shading stage evaluates physically based materials under lights and environment maps, followed by tone mapping and temporal antialiasing. The one idea that holds it together is that every stage is data-parallel and expressed as GPU work: culling and draw-list building are compute shaders that write indirect draw arguments, so the CPU only records a small, fixed command list.

Parts:

• Visibility — frustum, occlusion and cluster culling — frustum-occlusion-culling (no page yet), Bounding volume hierarchy, homogeneous-transforms (no page yet), image-pyramids (no page yet)
• Geometry and level of detail — meshlets, a LOD hierarchy, streaming — meshlet-cluster-lod (no page yet), mesh-simplification-quadric (no page yet), half-edge-mesh (no page yet), Graph partitioning, d-ary heap, LU and Cholesky factorization
• Draw submission — GPU-built draw lists and command recording — gpu-driven-indirect-rendering (no page yet), GPU compute shaders, parallel-prefix-sum (no page yet), GPU command buffers, gpu-synchronization-fences (no page yet), pipeline-state-caching (no page yet)
• Frame graph — passes, resource lifetimes, barriers — Render graph (frame graph), Topological sort, Interval tree
• Surface shading — materials and image-based light — Physically based shading (microfacet BRDF), image-based-lighting (no page yet), mipmapping-texture-filtering (no page yet), importance-sampling (no page yet), spherical-harmonic-transform (no page yet), texture-compression-bc-astc (no page yet)
• Lighting and shadows — many lights, shadow maps, depth precision — clustered-deferred-shading (no page yet), shadow-mapping (no page yet), depth-buffering-reverse-z (no page yet), triangle-rasterization (no page yet)
• Resolve and post — colour pipeline and temporal antialiasing — Color space conversion, Tone mapping (HDR to display range), temporal-antialiasing (no page yet), Image resampling filters, low-discrepancy-sequences (no page yet), convolution-image-filters (no page yet), Quaternion rotations

 CPU (small)                           GPU (does the scene-sized work)
 scene update ─► instance/transform buffers ─► ┌─ cull pass 1: instances, clusters vs frustum + last-frame HZB
 frame graph compile ─► command buffers ─────► │      │ writes visible list + indirect args (prefix sums)
   (passes, barriers,                          │      ▼
    transient memory)                          │  depth/visibility pass ─► build HZB ─► cull pass 2 (disoccluded)
                                               │      ▼
                                               │  shadow passes (cascades, lights)
                                               │      ▼
                                               │  light culling: lights ─► clusters (froxels)
                                               │      ▼
                                               │  shading: PBR + IBL + shadows ─► HDR colour
                                               │      ▼
                                               └─ post: TAA ─► bloom/DoF ─► tone map ─► swapchain

How the parts fit together

Visibility

frustum-occlusion-culling (no page yet) tests bounds against the six frustum planes and against a depth pyramid. Bounds come from a Bounding volume hierarchy so whole subtrees are rejected at once; the test needs homogeneous-transforms (no page yet) to take a bound to clip space and image-pyramids (no page yet) for the hierarchical Z-buffer (HZB), where each texel keeps the farthest depth of its footprint. On the GPU this is a GPU compute shader with one thread per instance or cluster.

The occlusion question has a chicken-and-egg problem: the depth buffer to test against does not exist until something is drawn. Nanite's answer, presented by Karis at SIGGRAPH 2021, is two-pass occlusion culling: test against the previous frame's HZB, draw what passes, build a fresh HZB from that, then test only what was rejected to find what became visible; the second pass is "usually a tiny fraction of the main pass", and frustum and LOD tests run once, in the first pass.

Contract to the next stage: a compact list of visible instances or clusters in a GPU buffer, plus a count.

Geometry and level of detail

For small scenes, discrete LODs per mesh are enough. For very dense geometry, meshlet-cluster-lod (no page yet) splits each mesh into small clusters and builds a hierarchy of coarser versions, so the choice of detail is made per cluster, not per object. In the Nanite talk a cluster is 128 triangles; each LOD level groups clusters, merges their triangles, simplifies to half the triangle count and splits the result back into clusters, repeating until one root cluster is left. Grouping which clusters to simplify together is Graph partitioning (the talk uses METIS), and the simplifier locks the shared boundary edges so neighbouring clusters at different levels do not leave cracks. Only a cut through the hierarchy needs to be resident, so geometry streams like virtual textures.

mesh-simplification-quadric (no page yet) is the standard way to collapse edges: Garland and Heckbert's quadric error metric ranks edge collapses and places the new vertex by solving a small linear system (LU and Cholesky factorization), with a priority queue (d-ary heap) on a connectivity structure such as half-edge-mesh (no page yet). I name this as the textbook method; the Nanite talk describes the build loop but I did not find its simplifier named there. The open library meshoptimizer provides meshopt_buildMeshlets, which balances vertex reuse inside a meshlet against culling efficiency (meshoptimizer README); Bevy's experimental meshlet module stores a BVH8 for culling and LOD selection.

Why clusters and not objects: culling at 64 to 128 triangles is fine-grained enough to reject hidden parts of big meshes, and a fixed small size gives the GPU uniform units of work. The price is a preprocessing step and a runtime that must be GPU-driven.

Draw submission

gpu-driven-indirect-rendering (no page yet) removes per-object draw calls. Haar and Aaltonen's 2015 pipeline culls instances, expands them into 64-triangle clusters, culls the clusters (frustum, occlusion, triangle backface), compacts the index buffer and issues a multi-draw indirect call; the CPU's job shrinks to coarse culling and batching. Stream compaction and the output offsets of each stage are typically a parallel-prefix-sum (no page yet). On the CPU side, GPU command buffers describes how the remaining recording is batched and how memory is reused across frames in flight; gpu-synchronization-fences (no page yet) orders GPU queues and CPU; pipeline-state-caching (no page yet) avoids the frame hitch when a shader permutation is compiled for the first time.

Contract: culling writes arguments in a buffer the draw call reads, so the CPU never reads back visibility. Reading it back would stall the pipeline for a full frame.

Frame graph

A modern frame has dozens of passes. The Render graph (frame graph) lets each pass declare what it reads and writes; per frame, the graph is compiled: unused passes are culled, a valid order comes from Topological sort, barriers are derived, and short-lived textures share memory when their lifetimes, held in an Interval tree or sorted ranges, do not overlap. The idea is documented in Yuriy O'Donnell's GDC 2017 talk on Frostbite's FrameGraph; Filament's source has a fg directory with FrameGraph.cpp and a DependencyGraph.cpp. The alternative, a hand-written fixed pass order, is simpler to read but turns every new effect into manual barrier and memory bookkeeping.

Surface shading

Physically based shading (microfacet BRDF) is the shared material model: a Cook-Torrance specular term and Lambertian diffuse, which Filament lists as its BRDFs. Light from the environment is image-based-lighting (no page yet): the specular part uses a prefiltered environment cubemap and a lookup table for the split-sum approximation (Karis, "Real Shading in Unreal Engine 4", 2013), the diffuse part an irradiance map or spherical harmonics (Filament design document). Offline tools do the filtering: importance-sampling (no page yet) builds the prefiltered levels, mipmapping-texture-filtering (no page yet) selects the level by roughness, and spherical-harmonic-transform (no page yet) compresses the diffuse irradiance into a few coefficients. Textures ship block-compressed, texture-compression-bc-astc (no page yet), so they stay compressed in GPU memory.

Lighting and shadows

Many lights are the classic scaling problem. Filament's design note compares the options: deferred shading supports many lights, but its G-buffer would use between 160 and 192 bits per pixel with its default material model, which means high memory bandwidth; plain forward shading handles multiple lights badly; tiled shading groups lights per screen tile but suffers at depth discontinuities. Filament therefore uses clustered forward shading: the view frustum is cut into a 3D grid of "froxels" with exponentially spaced depth slices, a compute shader assigns every light to the froxels it touches, and each fragment finds its froxel and loops over only those lights. This is the clustered-deferred-shading (no page yet) idea; the original paper by Olsson, Billeter and Assarsson (2012) reports, for one scene of around 2,400 lights, 17 ms against 26 ms for tiled shading, and the numbers are specific to that scene and hardware.

Choice between the variants:

• Clustered forward (Filament; Godot's Forward+ renderer; Bevy has a cluster module in bevy_pbr): keeps MSAA and transparency simple, low bandwidth.
• Deferred or visibility buffer (Bevy has a deferred module; Nanite shades from a visibility buffer of object and triangle ids): shading cost depends on pixels, not on triangles, which suits dense geometry. Nanite's documentation lists forward rendering and MSAA among unsupported features.
• Single-pass forward (Godot Mobile and Compatibility renderers): cheapest base cost on weak hardware.

shadow-mapping (no page yet) renders depth from each light, with cascades for the sun and filtering such as PCF; depth-buffering-reverse-z (no page yet) gives the depth precision that makes huge view distances and shadow cascades usable. triangle-rasterization (no page yet) is the fixed-function core under every pass, and at micro-triangle density it becomes the bottleneck: Karis reports a software rasterizer in compute about 3x faster than hardware on average for Nanite's small triangles.

Resolve and post

Lighting is computed in linear HDR. Color space conversion moves between the transfer curves, and Tone mapping (HDR to display range) compresses to the display; Filament lists several tone mappers. temporal-antialiasing (no page yet) jitters the projection each frame using a low-discrepancy-sequences (no page yet) pattern, reprojects the previous frame with motion vectors, and clamps the history to limit ghosting; resolve and upscale filters come from Image resampling filters, and bloom and depth of field from convolution-image-filters (no page yet). Quaternion rotations represent camera and object orientation in the scene update.

Data and control flow

1. Import a dense mesh (offline)

1. Cluster triangles into groups of 128 and group nearby clusters.
2. Merge, simplify to half the triangles with locked boundaries, split into clusters; repeat to a root.
3. Store the hierarchy and bounds; stream pages on demand at runtime.

2. Render a frame

1. CPU: update transforms and visible-set inputs; compile the frame graph; record command buffers.
2. GPU, cull pass 1: instances then clusters against frustum and last frame's HZB; write the visible list and indirect arguments.
3. Draw visible clusters into a depth or visibility target; build the HZB.
4. Cull pass 2 for what was occluded last frame; draw it.
5. Shadow passes for each light that needs one.
6. Assign lights to froxels; shade opaque pixels, then transparent ones.
7. Post: TAA, bloom, depth of field, tone map, present. Most frame time is in geometry and shading passes, and the culling passes are cheap by design. The CPU should finish its part well before the GPU does, otherwise frames in flight stall.

3. Compile a frame graph Collect pass declarations, cull passes whose outputs nobody reads, sort, assign memory to transient resources by lifetime, insert barriers, record.

Scaling

Limits in the order you reach them:

1. CPU draw cost with many objects: batch, then move culling and draw-list building to the GPU.
2. Overdraw and geometry cost with detailed meshes: occlusion culling, cluster LOD, a visibility buffer.
3. Lighting cost with many lights: froxel light lists, cheaper shadow methods for most lights, baked lighting for static parts.
4. Memory and bandwidth: block-compressed textures, streaming of geometry and textures, transient resource aliasing, fewer full-resolution targets.
5. Resolution: render below native size and upscale temporally (Filament supports dynamic resolution with FSR).
6. Shader permutations: hitches on first use; precompile and cache pipelines.

Failure modes

Real implementations

Sources

• B. Karis, R. Stubbe, G. Wihlidal, Nanite: A Deep Dive, SIGGRAPH 2021 Advances in Real-Time Rendering. slides
• U. Haar and S. Aaltonen, GPU-Driven Rendering Pipelines, SIGGRAPH 2015 Advances in Real-Time Rendering. slides
• O. Olsson, M. Billeter, U. Assarsson, Clustered Deferred and Forward Shading, High Performance Graphics 2012. preprint
• Y. O'Donnell, FrameGraph: Extensible Rendering Architecture in Frostbite, GDC 2017. GDC Vault
• B. Karis, Real Shading in Unreal Engine 4, SIGGRAPH 2013 course notes. notes
• M. Garland and P. Heckbert, Surface simplification using quadric error metrics, SIGGRAPH 1997. DOI record
• Google, Filament design document and README.
• Godot, Renderers; Epic Games, Nanite virtualized geometry; meshoptimizer README.