autonomous-trucking

How it is built

Architecture overview

An autonomous truck replaces the driver of a heavy articulated vehicle on a fixed set of highway routes between freight terminals. Compared with a robotaxi it has fewer scenarios (a limited road type, hub to hub) but harder physics: it needs a long stopping distance, so it must see and predict far ahead, and a trailer behind the cab changes how it turns, brakes and recovers. The software is a pipeline run in a loop: sense with cameras, radar, lidar, inertial and satellite sensors; localize against a prebuilt map; perceive objects and track them; predict what each will do; plan a trajectory; control steering, throttle and brakes. Beside the pipeline sits an independent safety layer that watches health, and if anything fails drives the truck to a minimal risk condition (usually a pullover). The one idea: the learned parts (detection, prediction, parts of planning) are allowed to be imperfect only because they are boxed in by an operational design domain, hard rules, redundant hardware and an independent monitor. Aurora is the documented commercial example; the open stacks are Apollo and Autoware.

Parts:

• Time and sensor geometry — one clock for all sensors and known sensor poses — Clock synchronization, camera-calibration (no page yet)
• Localization and map — where the truck is, in a map of lanes and signs — Extended Kalman filter, sensor-fusion (no page yet), point-cloud-registration-icp (no page yet), slam (no page yet), R-tree
• Perception — objects and their motion from raw sensors — Convolutional networks, Object detection, Multi-object tracking, interacting-multiple-model-filter (no page yet)
• Prediction — likely futures of each road user — trajectory-prediction (no page yet), Transformer block
• Planning — a safe, comfortable trajectory — motion-planning (no page yet)
• Control — follow the trajectory with the truck's actuators — Model predictive control
• Compute platform — move data between processes, fit the networks on the computer — shared-memory-ipc (no page yet), Weight quantization
• Safety layer — detect faults, hold a safe state, argue it is safe enough — safety-monitor-architecture (no page yet), Fault tree analysis

 cameras ─┐                      ┌── map (lanes, signs) ◄── offline mapping (SLAM)
 radars  ─┤  time-stamped        ▼
 lidars  ─┼─► sync ─► localization (EKF + scan matching) ─► pose
 IMU/GNSS ┘     │                                               │
                ├─► detection ─► tracking ─► prediction ─► motion planning ─► MPC ─► steer/brake/throttle
                │                                (rules + learned)      ▲        ▲
                └── health ──► safety monitor (independent) ─────────────┴────────┘ minimal-risk maneuver
                                         │
                                  remote assistance (human, not in the loop)

How the parts fit together

Time and sensor geometry

Fusion is only as good as its timestamps: at 75 mph (about 33 m/s) a 50 ms skew between lidar and camera moves the same object by about 1.7 m. Clock synchronization disciplines sensors and computers to one time base (hardware-timestamped PTP, per the block page, gets under a microsecond; GNSS time can be the root) and each measurement carries the time it was taken, not received. camera-calibration (no page yet) gives each camera its intrinsics and the pose relative to the vehicle (Zhang's method is the classic); extrinsics between lidar, radar and cameras are calibrated the same way and re-checked, because a truck flexes and vibrates more than a car. Contract: every message downstream is (sensor id, capture time, data) and the sensor-to-vehicle transform is versioned.

Localization and map

The truck needs its pose to a fraction of a lane width, more precise than GNSS alone gives under bridges or in cuts. sensor-fusion (no page yet) combines the motion model (IMU, wheel speed) with GNSS and with a match of the current lidar scan against the map. The estimator is an Extended Kalman filter because vehicle motion and the scan-matching measurement are nonlinear; the linear Kalman filter is its base case. The scan match is point-cloud-registration-icp (no page yet): ICP or NDT align the scan to the map and return a pose with a covariance the filter can weigh. The map itself is built offline with slam (no page yet), then curated by hand and fleet reports; Aurora's report describes a proprietary map that the fleet updates when a lane is closed (source: Aurora Driverless Safety Report 2025). At run time the planner asks "which lane elements are within X metres of this pose" and that is a range query in an R-tree of lane geometry, not a scan. The map is a dependency with a version; an out-of-date map is a failure mode of its own.

Perception

Cameras, radar and lidar see differently: cameras give texture and colour, lidar gives geometry, radar gives velocity and keeps working in rain and fog. Aurora uses all three and its FMCW lidar (FirstLight) is described as detecting objects at more than 450 m and measuring their relative velocity directly (source: Aurora Driverless Safety Report 2025); that range is the answer to trucks' stopping distance. Convolutional networks are still the compute-efficient backbone for images and point-cloud encoders (PointPillars is a well-known lidar example); transformers are the alternative backbone when compute allows. Object detection outputs scored, classified 3D boxes per frame, with non-maximum suppression or a set-prediction head removing duplicates. Multi-object tracking turns per-frame boxes into persistent identities: predict each track with a Kalman filter, associate with the Hungarian assignment, and manage births and deaths (the SORT paper is the compact reference). Vehicles that brake or change lanes break a single constant-velocity model, so the interacting-multiple-model-filter (no page yet) runs several motion models in parallel and weights them by how well each explains the data. Contract with prediction: a list of tracks with state, covariance, class and age, at the sensor rate.

Prediction

trajectory-prediction (no page yet) gives for each tracked road user several possible futures with probabilities, conditioned on history and the map (lane graph). The usual network is attention over agents and map elements, so it is built from Transformer blocks; Motion Transformer (MTR) is a published example of a multi-modal predictor. A recurrent network is the cheaper alternative. The planner consumes these futures as obstacles in space and time; for a truck the horizon must be long, because a slow reaction to a lane change far ahead is expensive. Probabilities are not truths: the planner has to stay safe against the unlikely futures too.

Planning

motion-planning (no page yet) searches for a trajectory that is collision-free against the predicted futures, obeys the lane and speed rules, respects the truck's kinematics (including the trailer's swept path) and is comfortable. Families: lattice or hybrid-A* search, sampling methods, or trajectory optimization; the geometry tests use k-d trees and collision checks on shapes. Aurora's report describes combining a learned planner with formal rules encoded as "invariants" (for example, stop at a red light) so the behavior is predictable and checkable (source: Aurora Driverless Safety Report 2025); that is the practical answer to "how do you certify a learned component". Contract with control: a time-parameterized path at a fixed rate, replaced every cycle, with the previous plan as a warm start so that it does not jump.

Control

Model predictive control is the usual choice for the truck's longitudinal and lateral control, rather than a PID controller alone, because there are constraints (steering rate, brake capacity, trailer stability) and several coupled inputs. Each tick it solves a small optimization (quadratic-programming, no page yet) over a short horizon using a vehicle model, sends only the first command and repeats. PID loops remain in the actuators below it (brake pressure, steering torque). Contract: a fixed-rate control loop with a deadline; a late or missing command is itself a fault that the safety layer sees. The plan is replaced as new perception arrives, so control follows a moving target.

Compute platform

A dozen processes exchange large messages (point clouds, images, tracks) at sensor rate. Copying them through sockets wastes the memory bandwidth the networks need, so the middleware uses shared-memory-ipc (no page yet), as in iceoryx's zero-copy publish/subscribe, with single-producer rings (SPSC ring buffers) for the sensor driver to first consumer hop. The networks run under a power and thermal budget in the cab, so they are quantized (Weight quantization) and validated on the exact deployed numerics, not the training ones. An end-to-end latency budget is set from sensor to actuator; every process has a deadline and reports misses.

Safety layer

The main computer is allowed to be complicated; the part that decides "is it still safe" must be simple. safety-monitor-architecture (no page yet) puts an independent checker next to the doer: it validates the plan against hard limits, watches heartbeats and health, and can force a safe state. A watchdog timer catches hangs; a memory protection unit isolates the monitor from the planner's bugs; hot-standby redundancy keeps a second channel current for the safety-critical functions. Aurora lists redundant steering, braking, power, motion management, communication and computing (source: Aurora Driverless Safety Report 2025). When a fault is found, the fault management system maps it to a predetermined response, and the planner generates the trajectory to a minimal risk condition, preferably a designated pullover spot off the highway rather than the shoulder (same source). To decide what must be protected, Fault tree analysis writes "truck leaves lane uncontrolled" as AND/OR logic over component failures and exposes single points of failure; it feeds the safety case, a structured argument with evidence (Aurora publishes its safety case framework).

Data and control flow

One drive cycle (order of 10 Hz; each stage has a deadline):

1. Sensors capture; drivers stamp each frame with the common clock and publish it into shared memory.
2. Localization predicts the pose from IMU and wheel speed, then corrects with GNSS and a scan match against the map tiles loaded around the truck; it publishes pose and covariance.
3. Detection runs per sensor or fused; tracking associates the detections to existing tracks, updates their filters and publishes tracks.
4. Prediction takes tracks and the lane graph (an R-tree query around the pose) and returns multimodal futures.
5. Planning builds candidate trajectories, scores them against rules and predicted futures, picks one and publishes it; the safety monitor independently checks it against hard limits.
6. Control solves the MPC problem against the current state and sends actuator commands; actuators report back and any missed deadline raises a fault.
7. Logging writes the inputs and outputs of every stage so the cycle can be replayed in simulation; the time goes to detection and prediction networks first, then planning.

One fault (loss of a sensor, say):

1. A health check fails; the monitor reads the failure's severity against a table built from the fault tree.
2. If the capability is degraded but sufficient, speed is limited and remote assistance is asked; if not, the planner receives a minimal-risk goal (preferred pullover if reachable).
3. The truck executes the maneuver and stops; it resumes only when the fault clears and a remote specialist or the system's own checks allow.

Scaling

1. Perception range and quality limit speed first: a truck can only go as fast as it can stop inside what it sees. Levers: longer-range sensors, fusion, better detectors, then lower speed in a constrained ODD.
2. Onboard compute and power. Levers: quantization, a smaller backbone, fewer cameras at full rate, pipelining stages, moving rarely needed models to event-triggered execution.
3. Long tail of scenarios. Levers: replay of logged drives and simulation, mining of hard cases from the fleet, retraining; a wider ODD only when the safety case closes for it.
4. Map coverage. Each new route needs mapping and validation. Lever: automated map building and fleet updates; route-by-route release.
5. Fleet operations: data upload, remote assistance staffing and terminal logistics scale with trucks; keep the on-truck system autonomous and put humans on exceptions only.

Failure modes

Real implementations

The alternative: one end-to-end network from sensors to trajectory skips hand-built interfaces and learns everything from data, but is harder to check, to constrain with rules and to argue safe; for freight the modular stack with explicit interfaces and an independent monitor is the documented choice.

Sources

• Aurora, Driverless Safety Report 2025 (perception, planning with invariants, mapping, fault management, redundancy, ODD). pdf
• Aurora, Safety Case Framework (public portion). framework; aurora.tech/safety
• A. Bewley et al., Simple Online and Realtime Tracking, 2016. paper
• A. Lang et al., PointPillars: Fast Encoders for Object Detection from Point Clouds, 2018. paper
• S. Shi et al., Motion Transformer with Global Intention Localization and Local Movement Refinement, 2022. paper
• Z. Zhang, A flexible new technique for camera calibration, IEEE TPAMI 2000.
• P. Biber and W. Strasser, The Normal Distributions Transform: A New Approach to Laser Scan Matching, IROS 2003.
• ISO 26262 (functional safety) and ISO 21448 (SOTIF), named in the Aurora report; UL 4600 for autonomous-product safety cases.
• Apollo, Autoware, iceoryx and openpilot repositories as linked above.