autonomous-driving

How it is built

Architecture overview

An autonomous driving stack reads sensors, works out where the vehicle is and what is around it, predicts what other road users will do, plans a trajectory that is safe, legal and comfortable, and turns it into steering, throttle and brake commands, many times per second. The classic structure is a layered pipeline: Autoware, the open reference, names seven stacks (sensing, map, localization, perception, planning, control, vehicle interface) and says the layering exists to make each module's role clear and replaceable (source: Autoware architecture documentation). Each layer consumes a typed, time-stamped message from the layer below and publishes one for the layer above. The idea that makes this work is time as a first-class quantity: every measurement carries a timestamp from a synchronised clock, every filter predicts to the measurement's time, and every stage has a latency budget, because a perfect answer that arrives 300 ms late describes a world that no longer exists. A second idea is that the control and safety layers stay simple and verifiable while the perception layers are learned and statistical. Open systems of this shape are Autoware, Baidu's Apollo and comma's openpilot; closed ones such as Waymo's share the structure but not the code.

Parts:

• Vehicle interface and timing — CAN in and out, deterministic scheduling, one clock — can-bus-framing (no page yet), real-time-linux-scheduling (no page yet), Clock synchronization
• State estimation — pose, velocity and a map around the vehicle — sensor-fusion (no page yet), Kalman filter, slam (no page yet), occupancy-grid (no page yet)
• Perception — objects and free space from camera and lidar — Object detection, bev-perception (no page yet), point-cloud-processing (no page yet), Multi-object tracking
• Prediction — likely futures of every agent — trajectory-prediction (no page yet)
• Planning — a collision-free trajectory through the predicted scene — motion-planning (no page yet), imitation-learning (no page yet)
• Control — follow the trajectory by actuator commands — Model predictive control, PID controller
• On-board compute — run the networks within the power and latency budget — Weight quantization, Kernel fusion

 cameras ─┐                         ┌─────────── map (HD vector map / point-cloud map) ─────────────┐
 lidar ───┼► time-stamp, sync ─► perception ──objects, free space──► prediction ──futures──► planning
 radar ───┤   (PTP / hardware       (detect → track)                                          │ trajectory
 GNSS/IMU ┤    timestamps)              ▲                                                      ▼
 wheel ───┘           │             localization (EKF/SLAM) ◄─ map                       control (MPC + PID)
 speed (CAN)          └──────► pose, velocity ────────────────────────────────►         │ steer, accel, brake
                                                                                          ▼
 vehicle  ◄── CAN frames ◄── vehicle interface (+ independent safety limits) ◄────────────┘
 monitor / watchdog: every stage has a deadline; missed deadline → degrade → minimal-risk stop

How the parts fit together

Vehicle interface and timing

The vehicle speaks CAN: short broadcast frames with an identifier that doubles as priority, resolved by bitwise arbitration on the wire (can-bus-framing, no page yet). A classical CAN data field is at most 8 bytes and CAN FD allows more (ISO 11898); Linux exposes both as sockets (source: kernel SocketCAN documentation). The interface layer translates between the stack's units (curvature, acceleration) and the vehicle's signals, and is the right place for a second, independent line of defence: comma's panda is a microcontroller that speaks CAN and CAN FD between the computer and the car and compiles vehicle-specific safety logic that limits what can be sent, regardless of what the main computer asks (source: panda README).

The computer must meet deadlines, not just be fast on average. real-time-linux-scheduling (no page yet) covers the tools: a PREEMPT_RT kernel, fixed-priority or deadline scheduling classes, pinned cores and locked memory so a page fault never lands in the control loop. The loop itself is a fixed-rate control loop. Autoware's own architecture document lists real-time processing, fail-safe and redundancy as work left out of the initial design, which is a useful reminder that a research stack and a production stack differ mostly here.

Clock synchronization gives all computers and sensors one timebase: NTP reaches about a millisecond, PTP with hardware timestamps under a microsecond (see that page). At 20 m/s a 1 ms error is 2 cm, and a 100 ms error between a lidar and a camera is 2 m, so the decisive requirement is that sensors stamp at the moment of measurement (hardware trigger or hardware timestamp), not when the driver reads the packet.

State estimation

sensor-fusion (no page yet) combines IMU, GNSS, wheel speed and perception-derived measurements into one pose and velocity with covariance. The core is a Kalman filter family estimator (usually an extended or error-state variant, see Extended Kalman filter) that predicts with the IMU at high rate and corrects with slower, noisier measurements, applying each at its own timestamp; the contract is a pose at a requested time, with an uncertainty. High-rate odometry feeds control, low-rate corrected pose feeds planning.

slam (no page yet) closes the loop when GNSS is poor: visual or lidar odometry accumulates drift, loop closure and pose-graph optimisation remove it (ORB-SLAM, LIO-SAM). Production stacks often avoid mapping at drive time and instead localise against a prior map by matching the live scan to it (point-cloud registration), keeping SLAM for building the map. An occupancy-grid (no page yet) is the shared free-space representation: each cell holds a log-odds probability of being occupied, updated from range returns; planners can query it cheaply.

Perception

Perception turns raw sensors into an object list and a free-space map.

• Lidar: point-cloud-processing (no page yet) discretises unordered points into voxels or vertical pillars so a convolutional network can run on them (PointPillars encodes a point cloud into a pseudo-image for a 2D backbone; Lang et al. 2018). Neighbour searches and ground removal use spatial structures such as the k-d tree and Spatial hashing.
• Cameras: bev-perception (no page yet) lifts multi-camera features into a bird's-eye-view grid: Lift-Splat-Shoot predicts a depth distribution per pixel, "lifts" features into 3D and "splats" them onto the grid (Philion and Fidler 2020); BEVFormer uses BEV queries that attend to camera features across space and time (Li et al. 2022). A BEV representation puts all cameras in one frame that planning can use directly. It needs camera calibration and transforms between sensor and vehicle frames.
• 2D detection: Object detection on single images remains the workhorse for signs, lights and lanes, and a cheap cross-check of the 3D path.
• Tracking: Multi-object tracking gives each object a persistent ID and a velocity from frame-to-frame detections (predict with a Kalman filter, gate, assign). Velocity comes from tracking, not from a single frame, which is why the tracker sits between perception and prediction.

The contract at the output is an object list with class, pose, size, velocity, covariance and timestamp, plus the free-space grid. Redundancy is deliberate: lidar, camera and radar fail differently, and late fusion of per-sensor detections degrades more gracefully than a single end-to-end network.

Prediction

trajectory-prediction (no page yet) maps each agent's history and the lane geometry to several possible futures with probabilities, because a car at a junction may go straight or turn and the planner must respect both. Representations range from rasterised scenes to vectorised maps and agents (VectorNet) and multi-modal heads using anchor trajectories (MultiPath) or attention networks over agents and map (Wayformer). Models are built from the attention blocks of Transformer block and sequence models; training uses the Loss functions for a mixture of futures. The contract with planning: for each agent, K trajectories with probabilities at fixed time steps over a horizon of seconds. The ego plan changes what others do, so a joint formulation (prediction conditioned on the planned ego trajectory) is better than treating the others as scenery.

Planning

motion-planning (no page yet) chooses a trajectory from a graph or a space of candidates: lattice and graph search (A* search, hybrid A* for vehicle kinematics), sampling planners such as RRT* (Karaman and Frazzoli 2011), or trajectory optimisation that minimises a cost over a time-parameterised path. Collision checks use geometric tests against the predicted agents' footprints (GJK and EPA for convex shapes) and the occupancy grid, and sampling planners rely on PRNGs (PCG, xoshiro). A common structure splits the problem: choose a path (geometry), then a speed profile along it, then smooth. The planner is replanned every cycle and must stay consistent with the previous plan; otherwise the car appears to hesitate.

imitation-learning (no page yet) is the learned alternative: behaviour cloning of a human driver's trajectories (Bojarski et al. 2016 trained a network from camera pixels to steering), with DAgger-style data collection to correct the distribution shift that arises when the learner drives itself into states the expert never showed (Ross et al. 2011). Planning-oriented end-to-end stacks such as UniAD train perception, prediction and planning as one differentiable system (Hu et al. 2023). Hybrid designs keep a learned proposal and a rule-based verifier; the verifier is what makes the safety case possible.

Control

Model predictive control solves, each tick, a small optimisation over a predicted horizon using a vehicle model (a kinematic or dynamic bicycle model integrated with, for example, Runge-Kutta 4), applies only the first input and repeats. It handles constraints (steering rate, lateral acceleration, comfort limits) and couples steering with speed. Its solver is a quadratic program (quadratic-programming, no page yet). Beneath it, PID controller loops run on single actuators with simple, well-understood behaviour: a steering-torque loop, a throttle/brake loop, an emergency fallback that does not depend on the planner. State feedback comes from the Kalman-filtered pose, so Kalman filter quality limits control quality. The output contract is a command per actuator at a fixed rate (order of tens of Hz to 100 Hz), within the actuator limits the vehicle interface enforces.

On-board compute

The networks must run on an embedded accelerator with a power budget. Weight quantization lowers weight and activation precision to cut memory traffic and raise throughput, with a measured accuracy cost per model. Kernel fusion removes round trips through memory between small operations; for perception pipelines, fusing preprocessing and normalisation matters as much as fusing layers. Both are deployment steps after training, and their results must be re-validated against the safety-relevant metrics, not just average accuracy.

Data and control flow

One planning cycle (order of 10 Hz):

1. Sensors capture; each frame is stamped by hardware at capture time. Messages move between modules on a middleware (ROS 2 in Autoware) with the stamp preserved.
2. Localization publishes the pose for the cycle's reference time, predicted forward with the IMU to that time.
3. Perception consumes the newest synchronised camera and lidar frames, produces 3D detections; the tracker associates them with existing tracks and outputs objects with velocities.
4. Prediction rolls out K futures per tracked agent, conditioned on the map and, ideally, the last ego plan.
5. Planning builds candidate trajectories, rejects those in collision with any predicted future above a risk threshold, scores the rest on safety, rules, progress and comfort, and publishes the best, stitched to the previous plan.
6. Control (separately, at a faster fixed rate) tracks the newest trajectory with MPC and PID.
7. The vehicle interface converts commands to CAN frames at the vehicle's rate; the independent limiter checks them. Time goes mostly to perception networks (GPU) and to the cycle's slowest stage; the budget is end-to-end from sensor timestamp to command.

A missed deadline:

1. A watchdog sees a stage has not published within its allowed age (or a message's timestamp is too old).
2. The consumer uses the last good output only inside a declared validity window (a few cycles), then marks it invalid.
3. The supervisor switches to a degraded mode (slower, larger margins), and if the condition persists, executes a minimal-risk manoeuvre (controlled stop in lane or at the roadside). The decision to stop does not depend on the module that failed.

Scaling

In the order the limits appear:

1. Perception quality is bound by data: the long tail of rare scenes. Levers: fleet data collection and mining, simulation for rare events (CARLA), auto-labelling, public datasets (nuScenes, Waymo Open Dataset) for benchmarking.
2. Latency is bound by the longest chain, usually perception. Levers: quantisation, fusion, smaller backbones, dropping frames under load, running independent sensors' networks in parallel, pipelining across cycles.
3. Compute and power are bound by the vehicle. Levers: dedicated accelerators, sharing one backbone across tasks, lower resolution for far fields of view.
4. Planning scalability grows with agents and branching. Levers: coarse-to-fine search, pruning agents by relevance, anytime planners.
5. Validation is the true bottleneck: proving safety needs miles or scenarios. Levers: scenario libraries, replay of logged drives (the same stack against recorded sensor data), simulation at scale, closed-loop regression on every change.

Failure modes

Real implementations

The alternatives: an end-to-end stack replaces the hand-built interfaces with one network from pixels to trajectory (UniAD, Bojarski); it removes information loss between modules but is harder to inspect and certify. A modular stack, as above, gives testable interfaces and a place for rule-based checks. A driver-assistance system (openpilot) keeps a human responsible and so can use a smaller sensor set and a weaker safety case than a driverless one. The robotaxi designs add HD maps, lidar redundancy and remote assistance.

Sources

• Autoware documentation, Autoware architecture (seven stacks, items left out of the initial design). docs
• commaai, openpilot and panda READMEs. openpilot, panda
• Linux kernel documentation: SocketCAN, real-time (PREEMPT_RT), deadline scheduling.
• A. Lang et al., PointPillars: Fast Encoders for Object Detection from Point Clouds, 2018. paper
• J. Philion, S. Fidler, Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D, 2020. paper
• Z. Li et al., BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers, 2022. paper
• J. Gao et al., VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation, 2020. paper
• Y. Chai et al., MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction, 2019. paper
• N. Nayakanti et al., Wayformer: Motion Forecasting via Simple & Efficient Attention Networks, 2022. paper
• S. Karaman, E. Frazzoli, Sampling-based Algorithms for Optimal Motion Planning, 2011. paper
• D. Dolgov et al., Path Planning for Autonomous Vehicles in Unknown Semi-structured Environments, Int. J. Robotics Research, 2010 (hybrid A*; no link).
• M. Bojarski et al., End to End Learning for Self-Driving Cars, 2016. paper
• S. Ross, G. Gordon, D. Bagnell, A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (DAgger), 2011. paper
• Y. Hu et al., Planning-oriented Autonomous Driving (UniAD), 2022. paper
• R. Mur-Artal, J. Montiel, J. Tardós, ORB-SLAM: a Versatile and Accurate Monocular SLAM System, 2015. paper
• T. Shan et al., LIO-SAM: Tightly-coupled Lidar Inertial Odometry via Smoothing and Mapping, 2020. paper
• H. Caesar et al., nuScenes: A multimodal dataset for autonomous driving, 2019. paper
• P. Sun et al., Scalability in Perception for Autonomous Driving: Waymo Open Dataset, 2019. paper