bare-metal-provisioning

How it is built

Architecture overview

A bare-metal provisioning system turns a rack of unconfigured servers into machines that a user can request, install and hand back, with the same ease as a cloud VM. The machine has no operating system and no agent to talk to, so everything starts from hardware that is already there: a baseboard management controller (BMC) that can power the box and choose its boot device, and a network card that can boot from the network. The system powers the machine on, makes it fetch a small boot program over DHCP and TFTP or HTTP, runs a temporary in-memory agent on it, lets that agent inspect the hardware, and later writes a disk image and reboots into it. A central store holds the desired state of every machine (what it should be running) and controllers drive each machine toward it, retrying and recovering as steps fail.

The one idea that makes it work is that the machine is treated as an unreliable remote device with a state machine: every step (power, boot, inspect, write, verify, reboot) is a small idempotent action that can fail, time out or be repeated, and the durable state of the whole fleet lives in one store, never in a process.

Parts:

• Inventory and addressing — which machines, which MACs, which BMC addresses, which IPs are free — ip-address-management (no page yet), built on longest-prefix-match (no page yet), radix-trie (no page yet), Roaring bitmaps
• Out-of-band control — power, boot device, sensors, serial console, over the BMC network — bmc-out-of-band-management (no page yet), SSH protocol, x509-certificates (no page yet)
• Network boot — the first code a machine runs, delivered by DHCP and TFTP/HTTP — pxe-network-boot (no page yet), x86-boot-process (no page yet), Secure boot chain
• Image deployment — streams, verifies and writes the OS image to disk, then configures first boot — image-based-provisioning (no page yet), Cryptographic hashes (SHA-2, BLAKE3), HTTP range requests, CRC32C checksums, fsync and direct I/O
• Controller and state store — holds desired and observed state of each machine and runs reconcile loops — Desired-state reconciliation, Watch notifications, MVCC, Message sequence numbers
• Work queue — long-running steps with retries and ownership — Durable job queue, Leases, Fencing tokens, Write-ahead log, Dead-letter queue, Idempotency keys, Exponential backoff with jitter
• Agent link — how the in-memory agent on the machine talks to the control plane — grpc-protocol (no page yet), Message sequence numbers

 operator / API ──► desired state store (MVCC, watch) ◄──── controllers (reconcile loops)
                           │ job queue (leases, retries)            │
              ┌────────────┴───────────┐                            │
              ▼                        ▼                            ▼
        IPAM / inventory         BMC driver ──(IPMI/Redfish)──► BMC ──► power, boot device
        (pools, MAC, BMC IP)                                          │
                                                                      ▼
   DHCP + TFTP/HTTP (iPXE) ◄──── PXE firmware on NIC ◄─────── machine powers on
              │ boot loader + kernel + initrd (signed)
              ▼
     in-memory agent ──gRPC/HTTP──► control plane: inventory, then "write image X"
              │ image over HTTP (range, hash check)
              ▼
     disk written, fsync ─► set boot device = disk ─► reboot ─► first-boot config

There are two lifecycles of one machine: enrolment (discover, inspect, benchmark, mark ready) and allocation (deploy, use, release, wipe). Both reuse the same boot path and the same agent.

How the parts fit together

Inventory and addressing

ip-address-management (no page yet) is the allocator for subnets, DHCP ranges and static addresses. A /24 or /16 pool is a bitmap of used addresses, and Roaring bitmaps keep large, sparse pools small and make "first free address in this range" a fast scan. Prefix lookups ("which subnet and VLAN does this address belong to?") use longest-prefix-match (no page yet) over a radix-trie (no page yet). The contract with the rest of the system is small: given a machine identity (MAC address or BMC-reported serial number), return its subnet, VLAN, lease and BMC address, and never hand the same address to two machines. That last guarantee is a conditional write on the allocation record (conditional-writes (no page yet), no page yet), not a lock in a process. DHCP leases themselves are time-limited by design, which is the same idea as Leases: an address ownership that expires unless renewed.

Out-of-band control

bmc-out-of-band-management (no page yet) is the only reason bare metal can be automated at all. The BMC is a small separate computer on the server board with its own network port and power, so it works when the main machine is off or broken. Two protocols matter: IPMI (older, binary, UDP) and Redfish (REST and JSON, defined by the DMTF). The provisioning system uses them for four verbs only: power on/off/cycle, set the next boot device (PXE once, then disk), read sensors and inventory, and open a serial-over-LAN console to see why a boot failed. Contract: the BMC is slow (seconds per call), flaky and a security risk, so it sits on a separate management network, calls go through one driver per vendor family, and every call is retried with backoff and jitter so a rack that powers back up after an outage does not hammer a few BMCs in lockstep. Credentials are secrets in a store, never in logs; Redfish uses TLS with x509-certificates (no page yet), and BMC shells and serial consoles are reached over SSH.

Network boot

pxe-network-boot (no page yet) is the bootstrap. The NIC firmware broadcasts a DHCP request that says "I am a PXE client of architecture X"; the answer carries an address plus the name and server of a boot file (see RFC 2131 for DHCP and RFC 4578 for the PXE-related options). Classic PXE downloads the file by TFTP, which is slow and has no integrity. The standard fix is a chain: the first file is iPXE, which then continues over HTTP with a script that the control plane generates per machine. That script is the control point: it can pick a kernel and initrd by MAC, run an installer, or tell an enrolled machine to "boot from local disk". The DHCP server must answer differently for BIOS, UEFI and already-iPXE clients, otherwise a machine loops between iPXE and the first loader.

What runs after the loader is the x86-boot-process (no page yet): firmware hands control to a loader, the loader to a kernel and initrd. When Secure Boot is on, each stage must verify the next (Secure boot chain); a provisioning system either ships a signed shim and kernel or turns the feature off for the ephemeral environment and enforces it again on the installed OS. Choose one policy for every machine.

Image deployment

image-based-provisioning (no page yet) is faster and more repeatable than running an installer on each node: take a prebuilt root filesystem image and write it. The in-memory agent downloads the image over HTTP, hashes it while streaming, writes it to the target disk, then configures first boot (hostname, network, SSH keys, cloud-init-style user data), fixes the boot loader, and sets the boot device to disk.

Blocks and why each is there:

• Cryptographic hashes — the image has a digest recorded when it is published; the agent refuses to boot a disk whose digest differs. SHA-256 is the interoperable choice; BLAKE3 is faster on a many-core machine, which matters for multi-GB images.
• HTTP range requests — an interrupted download resumes at the last byte instead of restarting; a cache or mirror in the rack can serve the same image to dozens of machines.
• CRC32C checksums — a cheap per-chunk check against corruption in transfer or on disk, hardware-accelerated on x86 and ARM; use it for chunk framing, not as a security check.
• fsync and direct I/O — "written" must mean durable before the agent reports success and reboots; skipping the flush gives machines that boot fine in the lab and lose their root filesystem after a power cut.

Contract with the controller: the agent receives an image reference plus digest, a target disk (by a stable identifier such as WWN, not /dev/sda) and a user-data blob, and reports a final state. It must be safe to re-run from the start, because a power loss in the middle is normal.

Controller and state store

Desired-state reconciliation is the heart of the control plane. Each machine has a spec (desired: image, network, owner) and a status (observed: power state, last step, last error). A controller loop compares them and issues the next small action. Idempotent steps replace long scripts that are lost when a process dies. The store behind it needs revisions: MVCC gives each write a version, and Watch notifications let controllers list at revision R and then watch from R+1 with no gap and no duplicate, so a restarted controller resumes exactly where it was.

Contract between neighbours: the user only edits the spec; controllers only write status (plus the next action in the queue); two controllers never act on the same machine at once (see the work queue). Machine state is a small explicit state machine (new, enrolling, ready, allocated, deploying, deployed, releasing, failed). Illegal transitions are rejected by the store with a conditional write. MAAS and Ironic both expose roughly this lifecycle to users (see their docs).

Work queue

Provisioning steps take minutes and talk to hardware that fails, so the controller does not call BMCs inline; it enqueues work. A Durable job queue persists each job, hands it to a worker under a Lease, and retries it if the lease expires. The sharp edge is the stuck worker: a worker whose lease expired may still be running and sends a power-off after a newer worker already started the install. Fencing tokens close that hole: each lease carries a growing number, and the BMC driver (or the state store) rejects an action with an older token than the latest one for that machine. Each action also carries an idempotency key, so a retried "set boot device to PXE" is the same call, not a second one. The queue's own log follows write-ahead log rules. Jobs that keep failing go to a dead-letter queue where an operator sees "machine R12-U14 failed to PXE three times, last BMC error X" instead of an invisible retry storm.

Agent link

The in-memory agent is a small client that has no state and no credentials except a one-time token baked into the iPXE script for that boot. It talks to the control plane over HTTPS or gRPC (grpc-protocol, no page yet), reports inventory (CPUs, memory, disks, NICs, firmware versions), pulls its next task and streams logs. Use message sequence numbers on the agent's report stream so the control plane can detect lost and duplicated messages after a flaky network or a controller restart. The link is pull-based on purpose: the machine reaches out, because the control plane cannot reach it until the OS is up.

Data and control flow

Enrolling a new machine:

1. An operator racks the machine, plugs in the BMC and data ports, and registers the BMC address and credentials (or leaves it to discovery).
2. The controller allocates a management IP and a PXE-boot lease; the BMC driver sets the next boot device to PXE and powers the machine on.
3. Firmware DHCP-boots iPXE; iPXE requests a script by MAC; the control plane returns the enrolment kernel and initrd.
4. The agent starts, reports hardware inventory, optionally runs memory and disk tests, and posts results.
5. The controller records inventory in status, labels the machine by hardware class, and marks it ready. The BMC powers it off.

Deploying a machine:

1. A user or scheduler writes a spec: image, network, SSH keys. The store bumps the revision; the controller's watch fires.
2. The controller picks a ready machine by a conditional write on its state (ready to allocated), so two requests cannot claim the same one.
3. A job is queued: set boot device to PXE once, power cycle. A worker takes it under a lease with a fencing token.
4. The machine boots the deployment agent through the same iPXE path. The agent downloads the image from a rack-local mirror, verifies the digest while writing, flushes, writes first-boot data and tells the controller it is done.
5. The controller sets the boot device to disk, reboots, and waits for the OS to call back (a phone-home request or a health probe) before marking the machine deployed. If it does not within the timeout, the job goes back to the queue with backoff, and after N attempts to the dead-letter queue.
6. On release the machine is wiped and returned to ready, or flagged for repair.

Scaling

In the order they usually bite:

1. Image transfer — hundreds of machines downloading multi-GB images from one server saturate its uplink. Levers: rack-local caches or mirrors, range-request resume, a compressed image, scheduling rolling batches rather than all at once.
2. DHCP and TFTP/HTTP boot servers — a datacenter power event makes every machine PXE-boot at the same moment. Levers: HTTP instead of TFTP, a DHCP relay per rack with local servers, and backoff with jitter on retries.
3. BMC concurrency — BMCs are slow and some fall over under parallel requests. Levers: a per-BMC concurrency limit of one in the driver, queue partitioning by rack, batching reads (inventory) with a long refresh interval.
4. Control-plane state — thousands of machines with frequent status writes. Levers: write status only on change, keep logs out of the state store, shard controllers by rack or site. A single database (MAAS uses PostgreSQL) goes a long way, but the watch stream should not be the bottleneck.
5. Multi-site — latency and partitions between sites. Lever: a regional controller close to each rack that works during a WAN outage and syncs desired state upward.

Boot time per machine is dominated by hardware POST and a reboot, not software; fleet-wide time is cut by parallelism across machines.

Failure modes

Real implementations

The alternative: do not provision metal yourself. Rent servers from a provider with a web API and an installer (hourly or monthly), and build only the layer above. That saves the BMC, DHCP and image machinery, at the price of provider lock-in, limited control over firmware, and a smaller choice of images and network layouts. Provision your own metal when you own the racks, need custom networking or firmware, or run enough machines that the provider's margin exceeds the cost of the team.

Sources

• DHCP: R. Droms, Dynamic Host Configuration Protocol, RFC 2131, 1997. RFC
• Y. Hu, M. Ni, D. Liu, Dynamic Host Configuration Protocol (DHCP) Options for the Intel Preboot eXecution Environment (PXE), RFC 4578, 2006: client system architecture and PXE DHCP options. RFC
• iPXE project documentation: chain loading
• DMTF Redfish standard: redfish.dmtf.org
• MAAS documentation: maas.io/docs
• OpenStack Ironic documentation: user guide
• Tinkerbell documentation: tinkerbell.org/docs