ar-runtime

How it is built

Architecture overview

An AR or XR runtime sits between the sensors and displays of a head-worn or hand-held device and the applications that want to draw into the user's world. It answers two questions continuously: where is the device in space, right now? and how do I get the application's frame onto the display so that it still looks locked to the world when the head has moved since the frame was rendered? The first is solved by visual-inertial odometry (VIO): a high-rate inertial sensor propagates the pose between camera frames, and camera features correct its drift. The second is solved by prediction and reprojection: the application renders for a predicted display time, and a compositor warps the finished frame with the newest pose just before scan-out. On top sits a spatial layer that exposes poses, reference spaces, planes and anchors through an API (ARKit, ARCore, or the Khronos OpenXR standard). The one idea that holds it together is that everything is timestamped: sensors, poses, predictions and frames all carry a time, and the runtime's job is to make those times line up at the photons.

Parts:

• Calibration — camera intrinsics, lens distortion, camera-to-IMU transform — camera-calibration (no page yet), PCA and SVD, levenberg-marquardt (no page yet), lens-distortion-correction (no page yet)
• Tracker (VIO front end) — feature tracking plus inertial propagation, outputs a timestamped pose — visual-inertial-odometry (no page yet), feature-detection-orb (no page yet), image-pyramids (no page yet), ransac (no page yet), Extended Kalman filter, Kalman filter, Explicit Euler, Robust statistics
• Map and optimizer — keyframes, 3D points, loop closure, global refinement — bundle-adjustment (no page yet), levenberg-marquardt (no page yet), LU and Cholesky factorization, Sparse matrices (CSR)
• Pose algebra — poses, spaces, anchors as relative transforms — Rigid-body transforms (SE(3) poses), Quaternion rotations, homogeneous-transforms (no page yet)
• Scene understanding — planes, depth fusion, meshes — tsdf-volume-fusion (no page yet), marching-cubes (no page yet), Spatial hashing, GPU compute shaders
• Frame loop and compositor — pose prediction, late pose sampling, warp, distortion, pacing — late-latching-reprojection (no page yet), image-warping-remap (no page yet), vsync-frame-pacing (no page yet), GPU command buffers, gpu-synchronization-fences (no page yet), real-time-linux-scheduling (no page yet)
• Foveation — shade fewer pixels where the eye is not looking — foveated-rendering (no page yet), Image resampling filters

 camera frames ──┐                       IMU samples (high rate) ──┐
                 ▼                                                ▼
          ┌─────────────────── tracker (VIO) ─────────────────────────┐
          │ undistort ─► features ─► match/track ─► fuse with IMU     │──► timestamped pose stream
          └─────────────┬─────────────────────────────────────────────┘        │
                        │ keyframes                                            │ predict(t)
                        ▼                                                      ▼
          ┌──────────────────────────┐   anchors, planes     ┌──────────────────────────────┐
          │ map: bundle adjustment,  │ ─────────────────────►│ runtime API (OpenXR / ARKit / │
          │ loop closure, depth fuse │                       │ ARCore): spaces, anchors      │
          └──────────────────────────┘                       └───────────────┬──────────────┘
                                                                              │ predictedDisplayTime, views
                                                                    ┌─────────▼─────────┐ layers
                                                                    │ application renders│───────┐
                                                                    └────────────────────┘       ▼
                       display ◄── scan-out ◄── distortion ◄── reproject with newest pose ◄─ compositor

How the parts fit together

Calibration

Everything downstream assumes the camera model is right. camera-calibration (no page yet) estimates intrinsics (focal lengths, principal point, distortion) and the extrinsic transform between camera and IMU; Zhang's method recovers a camera from several views of a planar pattern and is refined by nonlinear least squares (PCA and SVD solve its linear initial step; levenberg-marquardt (no page yet) does the refinement). The same distortion model has two uses: the tracker undistorts feature locations, and the compositor applies the inverse for the headset's lenses, which is lens-distortion-correction (no page yet): the rendered frame is pre-warped so that the optics unwarp it. A calibration error shows up later as a scale or drift problem in tracking that no filter tuning fixes.

Tracker

visual-inertial-odometry (no page yet) estimates the 6-DoF pose by fusing camera features with inertial measurements. ARCore describes this as SLAM that tracks visually distinct features in the camera image and combines them with IMU data to estimate pose over time (ARCore fundamentals). Two filtering styles exist:

• Filter-based: an Extended Kalman filter, which linearises the nonlinear motion and measurement models at the current estimate, carries the pose and a window of past camera poses; the multi-state constraint Kalman filter of Mourikis and Roumeliotis (2007) is the classic example. The Kalman filter page covers the linear core.
• Optimisation-based: a sliding window of keyframes solved by nonlinear least squares, as in VINS-Mono, which fuses pre-integrated IMU measurements and feature observations in a tightly coupled optimisation and adds loop detection and relocalisation (Qin et al.). Filters cost less per frame and suit constrained devices; optimisation is typically more accurate and gives a natural path to loop closure. Both rest on the same inputs.

Inertial data arrives much faster than images, and each camera frame needs the motion between two frames. Naively stepping the state with Explicit Euler drifts; the standard answer is to pre-integrate the IMU measurements between selected keyframes into single relative-motion constraints, which keeps the optimisation size bounded (Forster et al., on-manifold preintegration). The visual side is feature-detection-orb (no page yet): FAST corners with binary descriptors, matched by Hamming distance, found on an image-pyramids (no page yet) so features survive scale change. Wrong matches are the main danger, so every estimation step needs outlier rejection: ransac (no page yet) for model fitting and Robust statistics (Huber-style losses) inside the optimiser.

Contract: the tracker outputs a pose with a timestamp and a covariance or a tracking-quality flag. It never outputs "the pose now"; callers ask for the pose at a time, and the tracker extrapolates from the latest propagated state.

Map and optimizer

bundle-adjustment (no page yet) refines camera poses and 3D points together by minimising reprojection error with sparse nonlinear least squares, solved by levenberg-marquardt (no page yet): damped Gauss-Newton where each step solves a linear system. The system has block-sparse structure (Sparse matrices (CSR)) and is solved by a Cholesky-type factorisation (LU and Cholesky factorization), usually after eliminating the points. Basalt shows the pattern: it extracts non-linear factors from VIO and combines them with loop-closing constraints in bundle adjustment to get a globally consistent map (Usenko et al.). Why separate from the tracker: the tracker must finish within a frame and can only look at a window; the optimiser can run at a lower rate on a different thread and correct the whole trajectory.

Contract: when the map is refined, the pose of the world relative to the tracker's odometry frame changes. Applications must not hold raw world coordinates for long; they hold anchors.

Pose algebra and anchors

Rigid-body transforms (SE(3) poses) is the algebra the whole stack shares: a pose is a unit quaternion (Quaternion rotations) plus a translation, with compose, invert and interpolate. OpenXR exposes poses through spaces: xrLocateSpace returns the pose of one space in another at a given time, and the LOCAL reference space is a world-locked, gravity-aligned origin while VIEW is attached to the head (OpenXR 1.1 specification). The spec also notes that a projection layer containing world-locked content should use a world-locked space, because the compositor may reproject it, and that head-locked content such as a heads-up display should use the VIEW space for the best reprojection.

An anchor is the stable handle for a point in the world. The XR_EXT_spatial_anchor extension lets an application create anchors that the runtime tracks and adjusts "independent of all other spaces and anchors" so that they keep their mapping to the real world; the stated benefit is that virtual objects rendered relative to an anchor do not drift as the runtime's understanding improves. ARCore says the same: poses can change as it refines its understanding, an anchor keeps the object's position tracked, and its guidance is to reuse anchors, keep objects within 8 m of their anchor and detach unneeded ones to save CPU. The contract is therefore one-directional: the runtime owns the world estimate and moves anchors; the application owns content and places it relative to an anchor.

Scene understanding

Planes: ARCore groups clusters of feature points that lie on common horizontal or vertical surfaces into planes and reports their boundaries; textureless surfaces such as a white wall may not be detected (ARCore fundamentals). Fitting a plane to noisy points is ransac (no page yet) followed by refinement. OpenXR defines plane tracking as a spatial capability in XR_EXT_spatial_plane_tracking. For occlusion and physics, tsdf-volume-fusion (no page yet) integrates depth frames into a truncated signed distance volume (Curless and Levoy, 1996), stored in a sparse Spatial hashing grid, and marching-cubes (no page yet) extracts a triangle mesh (Lorensen and Cline, 1987). Fusion is data-parallel and runs as GPU compute shaders. Contract: scene understanding consumes tracked poses and publishes entities in the same spaces as anchors.

Frame loop and compositor

The OpenXR frame loop encodes the latency strategy. xrWaitFrame returns a predictedDisplayTime; the application uses that time when it calls xrLocateViews and xrLocateSpace, renders, and submits layers with xrEndFrame. The spec says an accurate predicted display time "is very important to avoid black pull-in by reprojection", that reprojection must not show an image before the display period it was predicted for, and that the better the prediction, the less latency the user sees. late-latching-reprojection (no page yet) is the compositor half: sample the newest head pose right before scan-out and warp the finished frame by the pose difference, which uses image-warping-remap (no page yet) to resample. When the application misses its frame budget, XR_EXT_frame_synthesis lets an app provide motion vectors and depth so the runtime can extrapolate a frame and stay smooth below the target rate.

vsync-frame-pacing (no page yet) keeps frames on the display cadence; the runtime may even shift the start of the app's frame interval relative to the refresh cycle to avoid GPU contention with the compositor (OpenXR spec). Work reaches the GPU as GPU command buffers, ordered between app, compositor and display by gpu-synchronization-fences (no page yet). On Linux-class devices the tracking and compositor threads need real-time-linux-scheduling (no page yet): fixed priorities, pinned cores and locked memory so that a missed deadline is not caused by the scheduler.

Foveation

With eye tracking, foveated-rendering (no page yet) lowers the shading rate away from the gaze point, then upsamples with Image resampling filters. OpenXR has vendor and EXT extensions for it (for example XR_FB_foveation, XR_META_foveation_eye_tracked, XR_VARJO_foveated_rendering). Gaze prediction matters, because the fovea moves faster than a frame.

Data and control flow

1. One tracking update

1. IMU samples arrive at a high rate; the state (pose, velocity, biases) is propagated by integration.
2. A camera frame arrives with its exposure timestamp; build an image pyramid, track existing features, detect new ones.
3. Reject outliers with ransac and a robust loss; update the filter, or add a keyframe factor to the sliding-window optimiser.
4. Publish a timestamped pose. The back end may adjust keyframes and points later (loop closure, bundle adjustment), which moves anchors. Time goes to feature extraction and the optimiser's linear solve; keeping it within one camera period is the design target.

2. One displayed frame (OpenXR)

1. xrWaitFrame blocks until the next frame interval and returns predictedDisplayTime.
2. xrBeginFrame; the app asks xrLocateViews for the eye poses and projections at that time. The runtime extrapolates the tracker's pose to that time.
3. The app renders each eye and calls xrEndFrame with projection layers (and depth, if offered).
4. The compositor waits for the right moment, reads the newest pose, reprojects each layer, applies lens correction and presents. Latency is the time from sensor to photon; the pipeline above trades it against smoothness. For scale: Apple states that its R1 chip, which processes input from 12 cameras, five sensors and six microphones, streams new images to the displays within 12 milliseconds (Apple newsroom, 2023). That is a vendor figure for the passthrough path.

3. Place and keep an object

1. The app raycasts against detected planes and takes the hit pose.
2. It creates an anchor at that pose; the runtime attaches it to the map.
3. Each frame the app renders the object at the anchor's current pose; when the map is corrected, the anchor moves and the object follows.

Scaling

Pull these levers in this order:

1. Latency and jitter first: better prediction, late pose sampling, a compositor on its own high-priority thread; this beats any amount of rendering speed.
2. Tracker cost: fewer features, smaller windows, pre-integration instead of high-rate optimisation variables, keyframe selection.
3. Map growth: marginalise or discard old keyframes, bound the number of anchors, keep the optimiser at a lower rate on another core.
4. Rendering load: foveated shading, dynamic resolution, frame synthesis when the app falls behind.
5. Power and heat on mobile devices: lower camera rate, run depth or fusion only when requested, ARCore's advice to drop unused anchors.

Failure modes

Real implementations

Sources

• Khronos, OpenXR 1.1 specification: frame timing (xrWaitFrame), reference spaces, XR_EXT_spatial_anchor, XR_EXT_spatial_plane_tracking, XR_EXT_frame_synthesis.
• Google, ARCore fundamental concepts and anchors.
• Monado, project page and README.
• C. Forster, L. Carlone, F. Dellaert, D. Scaramuzza, On-Manifold Preintegration for Real-Time Visual-Inertial Odometry, IEEE T-RO 2017. arXiv
• T. Qin, P. Li, S. Shen, VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator, IEEE T-RO 2018. arXiv
• C. Campos et al., ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM. arXiv
• V. Usenko et al., Visual-Inertial Mapping with Non-Linear Factor Recovery. arXiv
• A. Mourikis and S. Roumeliotis, A Multi-State Constraint Kalman Filter for Vision-aided Inertial Navigation, ICRA 2007. DOI record
• Z. Zhang, A flexible new technique for camera calibration, IEEE TPAMI 2000. DOI record
• B. Curless and M. Levoy, A volumetric method for building complex models from range images, SIGGRAPH 1996. DOI record
• W. Lorensen and H. Cline, Marching cubes: A high resolution 3D surface construction algorithm, SIGGRAPH 1987. DOI record
• Apple, Introducing Apple Vision Pro, newsroom, June 2023.