gigawatt-ai-campus

How it is built

Architecture overview

A gigawatt AI campus is one training job's machine, built as a building complex: tens to hundreds of thousands of accelerators on one low-latency network, fed by a power system the size of a small city and cooled by a plant that must keep every chip inside its thermal limit. The jobs are synchronous: every accelerator computes, then all of them exchange gradients, then repeat. That single fact shapes everything. The network must behave like one big switch, a single slow or dead GPU stalls thousands of others, the power draw swings in step across the whole site, and the useful output is measured as the share of wall-clock time that training really progresses. The idea that makes it work is to treat the campus as one control system with three coupled loops (power, heat, data) and one job scheduler on top, so a fault or a power swing in one loop is absorbed before it reaches the training job. Public examples are xAI's Colossus in Memphis, the OpenAI/Oracle Stargate site at Abilene and Meta's Llama training clusters (source: NVIDIA newsroom, 28 Oct 2024; Wikipedia, Colossus, Stargate LLC; Llama 3 paper).

Parts:

• Power chain — grid or onsite generation, UPS, distribution, battery buffers — ups-power-distribution (no page yet), battery-state-estimation (no page yet), droop-control (no page yet)
• Power management — budgets, capping and smoothing of the load — power-oversubscription-capping (no page yet), power-management-dvfs (no page yet)
• Cooling — heat removal from chip to outdoors, and its controller — datacenter-cooling (no page yet), PID controller, Model predictive control, finite-volume-method (no page yet)
• Telemetry and anomaly detection — per-second sensor streams from chips, racks and plant — Gorilla XOR compression, Time-series anomaly detection
• Machine management — out-of-band control of every server and health screening of every GPU — bmc-out-of-band-management (no page yet), gpu-health-checks (no page yet)
• Network fabric — topology, RDMA transport, congestion control — network-topologies (no page yet), rdma (no page yet), datacenter-congestion-control (no page yet)
• Collectives — the gradient exchange itself — Collective communication, Ring all-reduce, in-network-aggregation (no page yet)
• Scheduling and recovery — placement, gang start, checkpoints — vector-bin-packing (no page yet), gang-scheduling (no page yet), distributed-checkpointing (no page yet)

 grid / on-site gen ──► UPS + battery ──► rack power ───────┐
        ▲  droop / ramp limits        (smoothing, capping)  ▼
        │                                             ┌─ server (BMC, GPUs) ─┐
 chillers, CDUs ◄── heat ◄───────────────────────────│ 8 GPUs, NVLink       │
   ▲ PID / MPC                                        └──────┬───────────────┘
   │                                    RDMA fabric (leaf → spine → aggregation)
 telemetry ◄── power, temperature, GPU health ─────── congestion control, collectives
   │                                                         │
   └────────► anomaly detection ──► scheduler ──► drain, gang restart, checkpoint

There are two time scales in one site. The training loop runs in seconds with sub-second power and network dynamics. The facility runs in minutes to years: chiller response, firmware upgrades, hardware replacement. Most of the engineering is the interface between the two.

How the parts fit together

Power chain and the grid

Each GPU in the Llama 3 405B training cluster is rated at 700 W TDP (source: Llama 3 paper), so a server of eight draws several kilowatts before cooling and network overhead; in a GB200 server GPUs account for more than half the provisioned power (source: Choukse et al., 2025). Multiplied by the rack count that reaches hundreds of megawatts and then a gigawatt. The grid connection often arrives later than the buildings: for Colossus, Wikipedia reports an initially limited grid connection, with temporary gas turbines and Tesla Megapack batteries filling the gap. So the power chain is a mix of sources, each with its own control behaviour.

UPS power distribution (no page yet) is the layer that carries the load through short interruptions and conditions the supply to the racks. Droop control (no page yet) is how several generators or inverters share a load without a central controller: each lowers its frequency or voltage slightly as its output rises, so they settle at shares proportional to their ratings. A campus with on-site generation is a small grid, and the training job is a violently varying load on it. Batteries between the grid and the racks need a state estimate (charge, health, temperature); Battery state estimation (no page yet) supplies it, typically with a Kalman-type filter on voltage and current (see Kalman filter), so the smoothing system knows how much energy it can really absorb or release.

Power management: oversubscription, capping and smoothing

Two opposite problems sit here. The first is headroom. Fan, Weber and Barroso measured thousands of servers over six months and found a gap of 7 to 16 % between achieved and theoretical aggregate peak power at cluster level, growing to almost 40 % in whole datacenters, which suggests deploying more equipment than the nameplate budget allows (source: Fan et al., ISCA 2007). Power oversubscription and capping (no page yet) is the mechanism: plan on measured aggregate demand, and hold a capping layer ready to cut the load if the sum approaches the limit. Power management with DVFS (no page yet) is the knob a cap turns: lower frequency and voltage reduce power at a performance cost. Llama 3's authors note that hot mid-day weather made GPU dynamic frequency scaling cost 1 to 2 % of throughput (source: Llama 3 paper): the power and cooling loops leak into training speed.

The second problem is the swing. Synchronous training alternates a compute-heavy phase and a communication-heavy phase, and the compute phase draws much more power. When tens of thousands of GPUs do this together, the aggregate swings by tens of megawatts, and the Meta paper lists checkpoint waits and job start or stop as other synchronous events (source: Llama 3 paper). Choukse et al. show that at scale the swings reach tens or hundreds of megawatts and that their frequency, if it matches resonant modes of the grid and of turbine-generator shafts (they cite a 1 to 20 Hz band, with concern around 2 to 3 Hz), can damage equipment, so utilities specify ramp rates and frequency limits (source: Choukse et al., 2025). Their mitigations come in three layers: software that injects filler work to flatten the dip, GPU firmware with power floors and ramp limits, and rack-level energy storage that absorbs and releases the difference. The headroom of the first problem is thus largely gone for training sites: synchronised jobs do not average out the way independent web workloads do. Software and floors waste energy; storage costs hardware; sites use more than one layer.

Cooling and its control

Almost all electrical power ends up as heat, and the plant must remove it before any chip throttles. Datacenter cooling (no page yet) covers the path from chip to outdoors; efficiency is reported as power usage effectiveness (PUE), the ratio of total facility energy to the energy delivered to computing equipment (source: Wikipedia, Power usage effectiveness). Controllers close loops on supply temperature and flow. PID controller is the baseline for each loop (pump speed, valve position); Model predictive control can coordinate loops with coupled dynamics and a known disturbance, here the planned job schedule. Slow thermal mass is what makes prediction pay: the cooling plant can pre-cool before a known power rise. Finite volume method (no page yet) is the offline tool: computational fluid dynamics of airflow or coolant distribution, used to place racks and check hot spots before the building exists, and sometimes as a reduced model inside the predictive controller.

Telemetry and anomaly detection

Every chip, rack, pump and breaker emits metrics, often once per second or faster, and the stream is retained for post-mortems. Gorilla XOR compression is the time-series storage idea suited to it: consecutive floating-point values are XOR-ed and mostly zero bits are stored, so a slowly changing temperature costs a few bits per sample. Time-series anomaly detection turns the streams into alerts: a GPU slowly running hotter than its neighbours, a link whose error counter climbs, a pump whose power drifts. At this scale a rule per sensor is impossible, so detectors compare each device with its peers in the same rack or job.

Machine management and health

Each server has a baseboard management controller (BMC): a small computer on its own network that can power-cycle the host, read sensors and update firmware without the host's operating system. BMC out-of-band management (no page yet) is that layer; OpenBMC is a Linux distribution for such controllers and DMTF's Redfish is a standard management API (source: OpenBMC repo; DMTF Redfish developer hub). GPU health checks (no page yet) run before a job starts and on a schedule: NVIDIA's DCGM exposes health checks and diagnostics for GPU, board and environment (source: DCGM docs). The cost of skipping them is large. Meta's team reports 466 job interruptions in 54 days of Llama 3 405B pre-training, 419 unexpected, about 78 % attributed to confirmed or suspected hardware issues, and notes that a single straggler can slow thousands of other GPUs (source: Llama 3 paper). A separate study of two research clusters (4 million jobs, over 150 million A100 GPU hours in 11 months) finds large jobs most vulnerable to failure (source: Kokolis et al., 2024).

Network fabric

Training traffic is large, synchronised and loss-sensitive. Network topologies (no page yet) for these sites are multi-tier fat trees. Meta's RoCE cluster for Llama 3 has 16 GPUs in a rack, 192 racks per pod for 3,072 GPUs with full bisection bandwidth, and eight pods for a 24K-GPU cluster, with 400 Gbps links (source: Llama 3 paper). RDMA (no page yet) lets one GPU's network card write to another's memory without involving the CPU. Over IP networks it runs as RoCEv2, which relies on priority flow control to avoid loss; datacenter congestion control (no page yet) keeps that from stalling the whole fabric, as in DCQCN (source: Zhu et al., SIGCOMM 2015). Large training flows are few and fat, which defeats plain ECMP hashing; Meta added more entropy fields in the RoCE header to spread them (source: Llama 3 paper). NVIDIA reports that xAI's 100,000-GPU Colossus kept 95 % throughput across its three tiers with Spectrum-X, against about 60 % for standard Ethernet (source: NVIDIA newsroom). These are vendor figures; the point is that fabric tail latency is a first-order design target.

Collectives

The fabric exists to run Collective communication such as all-reduce, all-gather and reduce-scatter. Ring all-reduce moves each gradient around a ring in chunks so every link is busy and bandwidth per GPU stays constant as the ring grows; the Horovod paper popularised it for deep learning (source: Sergeev and Del Balso, 2018). Latency grows with ring length, so large jobs use trees, hierarchies and topology-aware layouts. Meta's collective library for Llama 3 is a fork of NCCL, and its parallelism dimensions are ordered so the chattiest traffic stays on the fastest links (source: Llama 3 paper). In-network aggregation (no page yet) moves the reduction into switches (SHArP, Graham et al. 2016), at the cost of switch features that tie you to a vendor.

Scheduling, gang start and checkpoints

A synchronous job needs all its GPUs or none. Gang scheduling (no page yet) starts all members together; in Kubernetes ecosystems Volcano has a "gang" plugin that checks whether the minimum pods can run, and Kueue manages quotas and admission (source: Volcano docs; Kueue docs). Vector bin packing (no page yet) places jobs on racks and pods: each job is a vector of GPUs, host memory and network, and fragmentation of one dimension strands the others. Placement is topology-aware so a job stays inside as few pods as possible.

Failures at the rate above make Distributed checkpointing (no page yet) essential. Meta's checkpoints save each GPU's model state, 1 MB to 4 GB per GPU, to a distributed file system (240 PB, 2 TB/s sustained, 7 TB/s peak) and the bursty writes were a challenge for the storage fabric (source: Llama 3 paper). Faster and more frequent checkpoints reduce lost work but also create synchronous power events, which closes the loop with the power section. Llama 3 reached over 90 % effective training time with automated maintenance, which caused at least one interruption daily (source: Llama 3 paper).

Data and control flow

One training step, seen from the campus:

1. Every GPU computes forward and backward passes on its shard; power rises toward TDP.
2. Gradients are reduced through the collective library over the RDMA fabric; GPUs idle during the exposed part and power drops.
3. The optimiser updates, the next step starts, and power rises again. The loop repeats at the step period.
4. The power layer sees the aggregate: cap and smoothing act on ramp rates, storage covers the difference.
5. Cooling responds slowly to the average, not the swing; telemetry samples all of it, and anomaly detection compares each GPU with its peers.

One failure, from fault to restart:

1. A GPU, link or host fails or becomes a straggler; the collective stalls or slows.
2. A watchdog in the training framework or the health check flags the member; the scheduler drains the host.
3. The BMC power-cycles or isolates it; a spare host is health-checked and reserved.
4. The job restarts as a gang from the latest checkpoint on the new set of hosts.
5. Lost work is the time since the last checkpoint plus restart.

Scaling

The binding constraint changes with size.

1. Time to power. At gigawatt scale the grid interconnect, not the chips, sets the schedule. Levers: onsite generation, batteries, staged energisation.
2. Power swing. Pull the three smoothing layers in the order of cost: software and firmware floors first, then storage.
3. Network tail latency. Levers: topology-aware placement, congestion control tuned to the fabric, entropy for load balancing, in-network reduction.
4. Failure rate. Mean time between interruptions falls with job size. Levers: health checks before and during runs, spares, faster checkpoints, straggler detection.
5. Cooling and density. Levers: predictive control, and CFD at design time.

Failure modes

Real implementations

The alternative: several smaller clusters in separate sites joined by a slower wide-area network. It avoids one gigawatt interconnect and one power point of failure, but synchronous training must then tolerate slow gradient exchange between sites. The single campus wins for one very large synchronous run; multi-site wins when power cannot be had in one place.

Sources

• A. Grattafiori et al., The Llama 3 Herd of Models, 2024. paper
• A. Kokolis et al., Revisiting Reliability in Large-Scale Machine Learning Research Clusters, 2024. paper
• E. Choukse et al., Power Stabilization for AI Training Datacenters, 2025. paper
• X. Fan, W.-D. Weber, L. A. Barroso, Power Provisioning for a Warehouse-sized Computer, ISCA 2007. doi
• Y. Zhu et al., Congestion Control for Large-Scale RDMA Deployments, SIGCOMM 2015. doi
• R. L. Graham et al., Scalable Hierarchical Aggregation Protocol (SHArP), COMHPC 2016. doi
• A. Sergeev and M. Del Balso, Horovod: fast and easy distributed deep learning in TensorFlow, 2018. paper
• NVIDIA, Ethernet Networking Accelerates World's Largest AI Supercomputer, Built by xAI, 28 Oct 2024. news
• Wikipedia: Colossus (supercomputer), Stargate LLC, Power usage effectiveness.
• Docs: NVIDIA DCGM, Volcano plugins, Kueue overview, Slurm gang scheduling, DMTF Redfish.