datacenter-facility

How it is built

Architecture overview

A datacenter facility turns grid power and outside air into a place where racks of servers run without stopping. Nearly all electricity entering the building leaves as heat, so the design is two pipes: a power train that delivers redundant electricity from the utility to the rack, and a cooling train that removes the same amount of heat. Around them sit a network fabric, a control and telemetry layer that watches meters, valves and breakers, and a fleet layer of software that places work, tolerates machine failures and keeps the power budget from being exceeded. The idea that makes it work (hyperscale case, after Barroso et al.): cheap capacity is bought by sharing the scarce resources (power headroom, cooling, uplink bandwidth) across many racks, and software absorbs the risk by being able to slow down, pause or move work within seconds.

Parts:

• Power train — utility feed, generators, UPS, PDUs, A/B redundancy — ups-power-distribution (no page yet), Hot-standby redundancy, battery-state-estimation (no page yet), modbus-protocol (no page yet)
• Power management — meters on every breaker level, controllers that cap servers before a breaker trips — power-oversubscription-capping (no page yet), priority-and-preemption (no page yet), power-management-dvfs (no page yet), Fixed-rate control loop
• Cooling train — containment, water loops, chillers or free cooling, liquid to the chip — datacenter-cooling (no page yet), PID control, Model predictive control, bacnet-protocol (no page yet)
• Thermal model — simulated airflow used to place racks and tune set-points — finite-volume-method (no page yet), Sparse direct solver, Sparse matrices (CSR), Graph partitioning
• Network fabric — top-of-rack switches under a multi-stage Clos of identical switches, BGP and ECMP — Clos fabric (no page yet)
• Telemetry and alarms — sensor streams stored compactly, turned into few actionable alarms — Alarm management, Time-series anomaly detection, Gorilla XOR compression, Delta encoding
• Fleet operations — job placement under power, space and network limits, failure handling, drains — vector-bin-packing (no page yet), Bin packing, Column generation, Primary-backup replication, Health checking, Graceful draining

 grid ─► substation ─► MV switchgear ─► UPS (A) ─► PDU (A) ─┐
              generators ──────────┘    UPS (B) ─► PDU (B) ─┼─► rack busbar ─► servers ─┐
                                                            │      (RAPL cap, sensors)  │ heat
                  power controllers ◄── meters (3 s) ───────┘                           ▼
                  (leaf → upper)  ──── cap / uncap ───────────────────────────► racks, hot aisle
                                                                                       │
 outside air / cooling tower ◄─ chillers ◄─ process water ◄─ fan coils / cold plates ◄─┘
          ▲                                  ▲
          └──── cooling controllers (PID / MPC) ◄── temperature, flow, valve position

 server ─ ToR switch ─ fabric switches (pod) ─ spine planes ─ edge ─ backbone      scheduler ─► placement
 (all parts) ──► telemetry bus ──► time-series store ──► alarms, anomaly detection, capacity reports

The facility changes slowly (cooling plant lasts around ten years, servers about three) while the IT load changes in seconds, so the design is about headroom and control loops.

How the parts fit together

Power train

Power enters at a utility substation that steps high voltage to medium voltage; unit substations step it to low voltage; the UPS in front of the floor combines a transfer switch between utility and generator, stored energy to bridge the gap, and power conditioning. A generator needs on the order of ten to fifteen seconds to start and take the load (Barroso et al.), which the UPS storage must cover. PDUs then split the feed into many small circuits, each with its own breaker so one shorted power supply trips only its own circuit. The topologies are the block ups-power-distribution (no page yet).

Redundancy is expressed as N+1 (survives one failure or one maintenance action), N+2, or 2N (a full A side and a full B side, each rack fed from both). The Uptime Institute's Tier ladder describes the same idea for the whole facility: Tier I a single path, II redundant components, III a concurrently maintainable alternate path, IV two simultaneously active paths; the lowest-rated subsystem sets the tier (Barroso et al.). It is the same idea as Hot-standby redundancy for controllers: a second source that is ready, and a switch that moves the load when the first goes silent.

Two design choices separate hyperscale from enterprise. Batteries move toward the load: a double-conversion UPS (AC to DC to AC) can waste as much as 15 % of the power through it, while newer designs and rack- or server-level batteries cut the loss to a few percent or less (Barroso et al.); the Open Compute Project's Rack & Power work goes the same way. Power is shared, not partitioned: Google's medium-voltage power plane lets many unit substations draw from one generator farm, so a storage row and a compute row can share capacity without stranding power.

Battery state of charge and health are estimated from voltage, current and temperature with a model: battery-state-estimation (no page yet) with an Extended Kalman filter. Meters and breakers are read over fieldbuses such as modbus-protocol (no page yet), polled at a fixed rate.

Power management: capping instead of overbuilding

A breaker trips when current exceeds its rating for a time inversely related to the overdraw: large spikes trip fast, small excesses are tolerated for minutes (racks and rack power panels sustained a 10 % overdraw for around 17 minutes at Meta; Wu et al.). That time budget is what makes software control possible.

The case for oversubscription is economic. Fan, Weber and Barroso observed up to 15 thousand servers for about six months: at cluster level the gap between achieved and theoretical peak power was 7-16 %, and almost 40 % for whole datacenters. In the book's summary of that data a rack reached 93 % of its peak at some point, but a 5,000-server cluster never exceeded 72 %, so 28 % of provisioned power would have been stranded and nearly 40 % more machines could have been hosted, if something guaranteed the budget.

Meta's Dynamo is the published example of that guarantee (block: power-oversubscription-capping (no page yet)). A hierarchy of controllers mirrors the power hierarchy:

• an agent on every server reads power (from a sensor, or a CPU-utilisation model on older machines) and applies a limit through Intel RAPL, which takes about two seconds to settle;
• a leaf controller per lowest-level power device pulls its agents every 3 seconds and runs a three-band algorithm: a capping threshold (typically 99 % of the breaker limit), a capping target 5 % below it, and a lower uncapping threshold, so the loop does not oscillate;
• a cut is taken from the lowest priority group first and, within a group, from the highest power bucket first (buckets of about 20 W), never below the group's SLA floor (the block is priority-and-preemption, no page yet);
• upper-level controllers hand "contractual" limits to the children that exceed their quota ("punish the offender first"); each controller applies the minimum of its own and its parent's decision.

In production Dynamo prevented 18 potential outages in six months and enabled an 8 % higher power utilisation in one data center (Wu et al.). The loop is a fixed-period controller (Fixed-rate control loop) with hysteresis; the actuator is the mechanism used for energy saving, power-management-dvfs (no page yet), though modern server CPUs have a narrow voltage range, so DVFS alone saves little (Barroso et al.). The safety valve for oversubscription: always run work that can be paused quickly (batch in a lower priority class), and do not provision so aggressively that the valve fires often.

Cooling train

All the heat has to reach the outside. A common layout has three loops (Barroso et al.): an air loop on the floor, a process water loop to the cooling plant, and a condenser loop that rejects heat through chillers and evaporative cooling towers. Fresh-air cooling is cheapest but needs filtering, depends on climate and reacts fast to faults because air stores little heat; two-loop systems are simpler but less efficient. Water economisers skip the compressor-driven chiller when the wet-bulb temperature is low enough; the block is datacenter-cooling (no page yet).

Airflow limits density: the book puts the practical raised-floor limit around 150-200 W per square foot without added cost. Containment separates hot and cold air so every server ingests the same temperature and the supply temperature can rise. Beyond that, in-row or in-rack heat exchangers and cold plates on the highest-power chips (Google's third-generation TPU cools four chips in series on one water loop) carry heat in water.

The control side: each chiller, pump and valve is regulated by a PID loop holding a temperature or flow set-point. Plant energy depends non-linearly on load, chillers running and weather, and the book notes that the large sensor data set makes machine-learning control of cooling attractive; Model predictive control is the classical form: predict the thermal response over a horizon, solve a small optimisation each tick with quadratic-programming (no page yet) or an Interior-point method, apply the first move. Cooling is safety-relevant: a facility can overheat within minutes without it, so generators back up the mechanical plant too (chillers and pumps can add 40 % or more to their load). Sensors and valves speak building protocols such as bacnet-protocol (no page yet).

Thermal model

Before racks are placed or a set-point changed, airflow is simulated: CFD on the room uses finite-volume-method (no page yet) and a Sparse direct solver over CSR matrices, after a Fill-reducing ordering built on Graph partitioning. It is too slow for control; it feeds layout decisions and tests control changes offline.

Network fabric

Bandwidth does not scale by buying a bigger switch: a switch chip has limited ports and power, so large networks cascade identical chips in a Clos / fat-tree (Al-Fares, Loukissas and Vahdat, SIGCOMM 2008, built from commodity switches). Cost per server rises with bisection bandwidth, so builders oversubscribe at the top-of-rack switch; the book says ratios of 4-10 are common.

Two production designs show the idea:

• Google Jupiter: a multi-stage Clos from merchant silicon, with centralised control that pushes one global configuration to all switches; capacity grew 100x over ten years to more than 1 Pbps of bisection bandwidth (Singh et al.).
• Meta data center fabric: small identical pods of 48 racks, four independent spine planes of up to 48 switches, every fabric switch connected to every spine in its plane, layer 3 with BGP4 only, ECMP flow hashing and a centralised BGP controller able to override paths. A spine plane is one failure domain that can be taken out of service without production impact (Meta engineering blog).

The shared point is uniformity: identical switches and pods make capacity grow by adding units and make the network programmable. The graph has no block yet for Clos fabrics, ECMP or switch operating systems.

Telemetry and alarms

Every stage reports slowly changing timestamped floats, the case Gorilla XOR compression and Delta encoding are made for. Turning them into action is two problems:

• Limits become operator-facing alarms with delay, deadband, acknowledgement and shelving (Alarm management); an alarm flood during a cooling fault hides the one that matters.
• Drift (a pump losing efficiency, PUE off its seasonal pattern) is caught by predicting the expected value and thresholding the residual (Time-series anomaly detection), built on STL decomposition, Robust statistics, Exponential smoothing and, for many sensors at once, Isolation forest.

The efficiency metric is PUE, facility power divided by IT power. Barroso et al. recommend real-time measurement and warn that published values differ in what overhead they count; annual averages are the comparable form. Google reports a fleet-wide trailing-twelve-month PUE of 1.09 for 2025, and quotes 1.54 as the average of respondents to the Uptime Institute's 2025 survey.

Fleet operations

At warehouse scale hardware reliability alone cannot deliver availability: servers with a 30-year MTBF in a cluster of 10,000 would still see about one failure per day (Barroso et al.), so software tolerates failures. In the studies the book cites, operator, configuration and software faults, not hardware, dominate service-level disruption.

The fleet layer uses the facility's limits as scheduling dimensions. Borg combines admission control, task packing, over-commitment and machine sharing, and spreads tasks to reduce correlated failures (Verma et al.). Placement is multi-dimensional packing (vector-bin-packing (no page yet), Bin packing, Column generation) over CPU, memory, rack power and fault domain. Replicated state uses Primary-backup replication; unhealthy machines leave the pool by Health checking and are emptied by Graceful draining.

Data and control flow

One power excursion (Dynamo-style capping):

1. A burst of requests raises server power on a rack row within seconds.
2. Each leaf controller pulls its agents (3 s cycle), sums them and compares the total with the breaker limit.
3. Above the capping threshold it computes the cut (distance to the capping target) and assigns it to the lowest priority group first, highest power bucket first.
4. Per-server limits go to the agents, which program RAPL; the effect settles in about two seconds.
5. The upper-level controller sees its aggregate fall; below the uncapping threshold, limits are removed.
6. The event goes to the time-series store; an operator is alerted if more than 20 % of a group's readings failed.

One cooling disturbance:

1. A chiller trips or the wet-bulb temperature rises; cooling controllers see supply temperature move and open valves, add a chiller or shift load from tower to chiller.
2. If it still climbs, alarm management raises one grouped alarm. Capping IT load through the actuator above is a natural design choice, not something the sources document. Water tanks buy time; air-cooled rooms have almost none.
3. Afterwards anomaly detection compares PUE and loop temperatures with their seasonal baselines.

One machine failure: health checks fail, the scheduler drains what it can, replicas serve the data, and the machine returns to the pool after repair.

Time goes: power control in seconds (3 s read, about 2 s to act), cooling in seconds to minutes.

Scaling

Limits in the order they bite:

1. Power capacity. Stranded power is expensive. Levers: mix workloads in one power domain, oversubscribe with capping and a pausable low-priority class, share capacity with a medium-voltage plane.
2. Cooling density. Levers: containment and higher supply temperature, in-row coolers, then cold plates and liquid loops for the hottest chips.
3. Network bisection bandwidth. Cost per server rises with fabric size. Levers: oversubscribe at the ToR, add spine switches to every plane, uniform pods, a centralised control plane.

Failure modes

Real implementations

The alternative: the enterprise or colocation design buys reliability in hardware and facility (2N power, Tier III or IV, redundant components in every server) and runs software that assumes the machine stays up. The hyperscale design buys it in software (replication, scheduling, capping) and builds a cheaper facility around that. Pick enterprise when the software cannot tolerate failures; pick hyperscale when the workload is large, replicated and tolerant.

Sources

• L. A. Barroso, U. Hölzle, P. Ranganathan, The Datacenter as a Computer: Designing Warehouse-Scale Machines, 3rd ed., 2018, open access. book. Chapters 3-5 and 7 used.
• X. Fan, W.-D. Weber, L. A. Barroso, Power provisioning for a warehouse-sized computer, ISCA 2007. abstract
• Q. Wu et al., Dynamo: Facebook's Data Center-Wide Power Management System, ISCA 2016. abstract and paper
• A. Singh et al., Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google's Datacenter Network, SIGCOMM 2015. abstract
• M. Al-Fares, A. Loukissas, A. Vahdat, A scalable, commodity data center network architecture, SIGCOMM 2008 (DOI 10.1145/1402958.1402967, checked in Crossref; paper not opened).
• A. Andreyev, Introducing data center fabric, the next-generation Facebook data center network, Engineering at Meta, 2014. post
• A. Verma et al., Large-scale cluster management at Google with Borg, EuroSys 2015. abstract
• Google Data Centers, Power usage effectiveness: page. Uptime Institute, Tier Classification System: page.
• Open Compute Project, Rack & Power. project page
• SONiC Foundation: sonicfoundation.dev; OpenBMC: openbmc.org.