← Back
Home › Systems › Cloud platform and compute

cloud-control-plane

Accepts API requests to create and change cloud resources, records the desired state durably and drives the fleet towards it.

EC2 API

Open the cloud-control-plane pagecloud-control-plane Cell based architecturedistributed: Independent full-stack cells with a thin router to bound blast radiusDesired-state reconciliationdistributed: Declare the target in a store; controllers repeatedly diff actual against desired from full state and apply idempotent fixes, so missed events only delay convergence.›Durable workflow executiondistributed: Ordinary code made crash-proof by logging every side-effect result and replaying the code from that log; activities run at least once, workflow logic must be deterministic.›Quorum replicationdistributed: Leaderless replication: write to W of N replicas, read from R; overlap (R + W > N) makes reads see the latest completed write.›Shuffle shardingdistributed: Assign each tenant a random subset of nodes to isolate failuresEvent sourcingfinance: Persist state changes as an append-only stream of immutable events per entity; current state is a fold over them, guarded by expected-version appends.›Idempotency keysfinance: Client-chosen unique key per logical operation, stored with a request fingerprint and the final response, so retries replay the first result instead of acting twice.›Ip address managementnetworking: CIDR allocation, subnet carving and lease tracking with bitmaps or prefix treesLong running operationsnetworking: Async APIs returning operation resources to poll or wait onPacket encapsulationnetworking: IP-in-IP, GRE, GUE, VXLAN and Geneve tunnelling headers and MTU effectsBin packingoptimization: Pack items of given sizes into the fewest fixed-capacity bins; sort descending, then first-fit, gives at most 11/9 of optimal and runs in O(n log n).›Authorization models (RBAC,ABAC, ReBAC)security: How to decide who may do what to which object; roles for coarse rules, relationships for sharing, and checks on every object.›Hmac request signingsecurity: HMAC canonical-request signatures and presigned URLs (SigV4 style) with expiryTamper-evident audit logsecurity: An append-only log whose Merkle root, signed per batch, lets anyone verify an entry is present and history was never rewritten, with log-size proofs.›Exponential backoff with jittersystems: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.›Token bucketsystems: A bucket of up to b tokens refilled at rate r; requests spend tokens, enforcing an average rate while allowing bursts of b.›Health checkingsystems: Decide per backend whether to route to it using probes and real-traffic errors, with consecutive-count hysteresis, capped ejection and a panic fallback.›Staged rolloutsystems: Rolling, blue-green and canary rollout with health gates and automatic rollbackWatch notificationsdistributed: A store numbers every write; clients list at revision R, then watch from R+1 and receive an ordered change stream with no gap, no duplicate, and a resume point.›Consistent hashingdistributed: Hash nodes and keys onto a ring; each key belongs to the next node clockwise, so membership changes move only about 1/n of keys.›Record replaydistributed: Record every nondeterministic input (time, randomness, I/O, model and tool outputs, per-tick player input) and re-execute deterministically for debugging, replays and state rebuildHierarchical timing wheelsystems: Timers bucketed by expiry into wheels of slots, coarser wheels for far deadlines: O(1) insert and cancel for millions of timeouts.›Anti-entropy repairdistributed: Replicas compare hash trees of a key range, descend only into differing subtrees and stream just those ranges, healing silent divergence in the background.›Hinted handoffdistributed: Park a write meant for a down replica as a tagged hint on a healthy node, replay it when the replica returns, and count it toward the write quorum only if availability matters more than consistency.›Vector clocksdistributed: One counter per node; comparing two vectors tells whether events are ordered or concurrent, which is exactly what conflict detection needs.›Fast hash functionsdata-structures: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.›Deterministic bucketingsystems: Hash a stable unit id with a per-purpose salt into a bucket; "in" means bucket below the percentage, so assignment is stateless, repeatable and only grows on ramp-up.›Durable job queuedistributed: Persisted tasks that workers lease for a visibility timeout, ack on success and retry with backoff; at-least-once execution that survives worker and queue crashes.›+ 13 moreBurn rate alertingsystems: Multi-window multi-burn-rate alerts on SLO error-budget consumptionTraffic splittingsystems: Weighted routing between versions for canaries and A/B, sticky by hashMessage sequence numbersdistributed: A per-stream counter on every message lets the receiver detect loss, duplicates and reordering, ack cheaply and resume after reconnect with "send from N".›Grpc protocolnetworking: GRPC framing over HTTP/2: unary and streaming RPCs, deadlines, statusMVCCstorage: Keep timestamped versions of each row so readers see a consistent snapshot without blocking writers; the default concurrency control in modern databases.›Slab allocatorsystems: Pre-carve pages into equal-size slots per size class with an in-place free list: O(1) alloc and free, no external fragmentation, cache-warm reuse.›Range partitioningdistributed: Cut the sorted key space into contiguous half-open ranges kept in a boundary map; split hot or big ranges, merge small ones, move ranges between nodes.›Phi accrual failure detectordistributed: Instead of a fixed heartbeat timeout, output a continuous suspicion level φ from the observed heartbeat interval distribution; adapts to each link's jitter.›Write-ahead logstorage: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›Dead-letter queuedistributed: A side queue for messages that keep failing: after N delivery attempts or a permanent error, park them with the reason so the main queue keeps moving.›+ 1 moreRollup aggregationstorage: Fold raw events into one mergeable partial aggregate per time bucket and dimension key, then merge fine buckets into coarser ones, so long ranges read few rows and stay exact.›Sliding-window counterssystems: Count events in the last W seconds with a ring of sub-bucket counters; the two-bucket weighted estimate gives rate limits in 16 bytes per key.›CRC32C checksumshardware-os: The Castagnoli CRC-32 with dedicated instructions on x86 (SSE4.2) and ARMv8. It is the standard fast integrity check for storage blocks, logs and network frames.›fsync, O_DSYNC and O_DIRECThardware-os: What fsync, fdatasync, O_DSYNC, O_DIRECT and F_FULLFSYNC really guarantee on Linux and macOS, and the exact call sequences for crash-safe files.›Hash aggregationstorage: GROUP BY via hash tables with partitioning and spill to diskTime partitioningstorage: Cut a table into fixed-interval time blocks so queries read only overlapping blocks and retention drops a whole block as one metadata change.›SIMD programminghardware-os: Process 4-16 lanes per instruction with AVX2, AVX-512 or NEON via Rust std::arch, with runtime dispatch and a portable fallback.›

How it is built

Architecture overview

A cloud control plane is the API through which customers create, change and delete resources: virtual machines, volumes, networks, addresses. A call such as RunInstances authenticates a signed request, checks permissions and quotas, records the customer's intent, and then drives a chain of slow, failure-prone steps (choose a host, allocate an address, attach a network interface, boot) that may take seconds or minutes and that must survive crashes in the middle. The data plane, which moves packets and disk blocks, does not run through it. The one idea that makes it work is that the API stores intent durably and returns early, and background machinery makes reality match the intent: every call is safe to retry, every long step is an operation the caller can poll, and every dependency has a bounded blast radius. This page follows the EC2 API and its AWS documentation, and compares OpenStack Nova, Azure Resource Manager and Google's API guidelines.

Neighbouring pages: Container orchestrator (the same reconcile idea over pods), Software-defined datacenter network, Durable workflow engine and Multi-tenant SaaS control plane.

Parts:

 client (SDK, signs request) ─► front end: verify signature → authorise → throttle → idempotency lookup
                                          │ accept: write intent (resource record, state=pending) + enqueue
                                          ▼ 202 / operation id (or the resource in "pending")
   resource store (quorum replicated) ◄── workflow / reconcile loop ──► placement (bin pack) ─► host agent
        │ every change is an event ──► audit log (signed batches)       ├─► IP address manager (subnet bitmaps)
        ▼                                                                └─► network mapping service ─► encapsulating dataplane
   client polls Describe… / operation  (eventually consistent reads)          cells: full copy of all of the above, tenants routed by key

How the parts fit together

API front end

Every call is signed. hmac-request-signing (no page yet) means the client builds a canonical form of the request (method, host, headers, body hash), then computes an HMAC-SHA256 with a key derived from its secret, and the server recomputes it (AWS Signature Version 4, whose string to sign includes a timestamp). The secret never travels and the signature covers the request, so it cannot be altered in flight. Once the caller is known, Authorization models (RBAC, ABAC, ReBAC) decide whether this principal may perform this action on this resource: cloud IAM is attribute- and policy-based, evaluated on every call, which is why the policy store must be quick and highly available. Then throttling: EC2 documents the Token bucket algorithm per account and action, with a bucket size that allows bursts and a refill rate for the steady state; calls over the limit get RequestLimitExceeded. Actions that change many resources also have a separate resource-token bucket: for RunInstances the documentation gives 1,000 tokens refilled at two per second. Counting windows (Sliding-window counters) are the cheaper alternative for coarse limits. The front end is stateless, so it scales by adding hosts, but limits and idempotency records must be shared state.

Request semantics

Clients time out, so every mutating call must be safe to repeat. Idempotency keys are EC2's client token: a unique string of up to 64 ASCII characters sent with RunInstances and similar actions; a retry with the same token and parameters succeeds without doing the work again, and a retry with the same token and different parameters fails with IdempotentParameterMismatch. Some actions (such as TerminateInstances) are idempotent by default. AWS's Builders' Library describes the same contract (a caller-provided client request identifier, with the created resource labelled by it). The server should record the token in the same transaction as the intent, or a crash between the two reintroduces the duplicate (this page's reading).

long-running-operations (no page yet) are the contract for calls that cannot finish inside a request. Google's API guidelines (AIP-151) say a slow method returns a google.longrunning.Operation instead of the final result, which the client polls or waits on; Azure Resource Manager returns 201 or 202 and a status URL in an Azure-AsyncOperation header, with Retry-After. EC2 takes a third route: the instance enters the pending state at once and the client polls Describe. In all three, reads are eventually consistent (EC2's documentation says a new resource is "eventually visible" and advises polling with exponential backoff), so a client that creates then immediately describes must expect a not-found. Waiting uses Long polling where offered, and retries use Exponential backoff with jitter on both sides so that a recovering service is not hit by every client at once. The accepted intent is queued on a Durable job queue.

Resource store and history

Quorum replication stores resource records without a single leader: write to W of N replicas, read from R, with R + W > N so reads see the last write; Vector clocks, Hinted handoff and Anti-entropy repair deal with divergence and temporary failures (the Dynamo design). A control plane often prefers a smaller, consistent store: Nova keeps its state in a SQL database, and AWS describes Physalia, a transactional key-value store that is highly available only for the keys each client needs, built as the configuration master for EBS and designed for cloud control planes. Whichever is chosen, concurrent updaters use Optimistic concurrency control: a version number checked on write.

Event sourcing keeps the history, not just the state: each resource is a stream of events (created, attached, resized, terminated) from which the current view is derived. The Azure architecture guidance names auditability and historical reconstruction as its benefits, and notes that a state-only store without a separate audit log loses history. The Tamper-evident audit log turns the events into evidence: AWS CloudTrail can hash each log file with SHA-256, write an hourly digest file signed with RSA, and include the previous digest's signature in the next so deletion or alteration breaks the chain (Digital signatures, Segmented log).

Orchestration

Desired-state reconciliation keeps the intent true over time: a record says "instance i-1 should be running on subnet s"; a controller compares it with the host's actual state and acts, again and again, so a crashed controller or a missed event only delays convergence. Nova shows the plumbing: REST API servers write the database, send RPC over a message queue to a scheduler, a conductor (which mediates database access and coordinates builds and resizes) and the compute service on each host. Watch notifications tell controllers what changed.

Creation is not a single state change but a sequence: reserve capacity, allocate an address, create an interface, start the host agent, record the result. Durable workflow execution runs such sequences as code whose progress is logged, so after a crash it resumes at the next step instead of repeating or abandoning the earlier ones; failed steps are retried with backoff and completed ones are compensated on rollback. Timers (waiting 30 s before the next check, timing out a step) are cheap with a Hierarchical timing wheel. The two ideas complement each other: workflows for the multi-step "do this once" path, reconciliation for "keep this true". See Durable workflow engine.

Placement and addressing

Bin packing picks the host. Nova's filter scheduler asks its Placement service, which tracks each host's inventory and usage, for candidates, then filters and weighs them. Microsoft's Protean paper describes the Azure equivalent: one instance serves an availability zone of 10,000 to 100,000 machines, keeps policy (a rule-based allocation agent) apart from mechanism, and uses multi-layer caching to answer in milliseconds. The contract with orchestration is that placement returns a reservation, not a promise: capacity may have been taken meanwhile, so the workflow retries on a conflict.

ip-address-management (no page yet) hands out addresses from subnets. EC2 documents that in every subnet the first four addresses and the last are reserved, which shows the shape of the problem: carve CIDR blocks without overlap, allocate and free single addresses fast, and never hand one out twice. A bitmap per subnet (Roaring bitmaps when sparse) and an atomic claim, guarded by a lease while the address is in transit, cover it. OpenStack Neutron exposes IPAM as a pluggable driver interface.

packet-encapsulation (no page yet) is what makes a customer's private network private on shared hardware. The host wraps each guest packet in an outer header addressed to the host that runs the destination guest, so customer addresses can overlap. Geneve (RFC 8926) is one standard for this; Google's Andromeda paper describes a control plane that can program connectivity for tens of thousands of VMs in seconds over a host dataplane. The control plane owns the mapping from virtual address to physical host; the dataplane only looks it up. See Software-defined datacenter network.

Blast-radius control

cell-based-architecture (no page yet) partitions the whole control plane into independent copies, each serving a slice of accounts or resources, with a thin router in front that keeps the partition key (AWS Well-Architected guidance). A bad deploy or an overloaded store hurts one cell. Nova calls the same idea "cells" for sharding a deployment. Health checking decides when a cell or host is out, and Consistent hashing and Deterministic bucketing assign keys and ramp changes stably. shuffle-sharding (no page yet) goes further for shared fleets: each customer gets a random small subset of workers, so a customer who sends poisonous requests exhausts only its own subset; with eight workers and two per customer there are 28 distinct shards, and AWS gives Route 53's 2,048 virtual name servers and shards of four as 730 billion. Cells fit whole-stack isolation and need a router, shuffle sharding fits shared pools and needs retrying clients. The Builders' Library "static stability" article defines the control plane as the machinery for changing resources and the data plane as what they do day to day; the design rule that follows is that running resources should not depend on the control plane.

Data and control flow

Launching an instance:

  1. The SDK signs RunInstances with a client token; the front end verifies the signature, authorises the principal, and takes one request token and many resource tokens.
  2. The idempotency record for the token is looked up; a repeat returns the earlier result.
  3. The resource records are written in state pending, with the intent event, and a workflow is started; the call returns.
  4. The workflow asks placement for a host, claims an address, creates the interface and tells the host agent to boot; each step is logged before the next.
  5. The mapping service publishes the virtual-address-to-host entry so the dataplane can encapsulate traffic to the new instance.
  6. The record moves to running; the event goes to the audit log; the client's Describe poll now returns it.

A retried call after a timeout: the client waits with jittered backoff, resends the same token, and receives the original result or the in-flight resource; no second instance is created.

A host failure: the health check marks the host down; the reconcile loop sees fewer instances than desired and starts the workflow again elsewhere, where policy allows.

Scaling

  1. API request rate. Levers: stateless front ends, per-action throttles, shed load early, cache authorisation and policy.
  2. Store size and write rate. Levers: partition by account or resource id, quorum tuning, move hot partitions, read replicas for Describe.
  3. Workflow count and placement. Levers: more workers, timer wheels, per-zone placement with cached inventory and reservation retry.
  4. Network programming. Levers: push only the mappings a host needs, gateways for rare flows, batched updates.
  5. Blast radius as the customer base grows. Levers: more and smaller cells, shuffle shards for shared services, staged deployment cell by cell.

Failure modes

FailureSymptomCauseFix
Duplicate resourcestwo instances for one callretry without a token, or token recorded after the workclient token stored atomically with the intent
Not-found right after createDescribe fails for a momenteventually consistent readretry with backoff, read from the leader when needed
Retry stormrecovery keeps failingall clients retry at oncejittered backoff, throttle with a clear error, shed load
Half-created resourcean address or interface with no instancecrash between workflow stepsdurable workflow with compensation, reconcile sweeper
Placement conflictlaunch fails after capacity was "found"two requests chose the same hostreservation with retry, optimistic versioning
Address collisiontwo guests with one IPracy allocation or stale leaseatomic claim, leases, audit of the allocation table
Cell-wide outageall customers in one cell failbad deploy or shared dependencystaged rollout cell by cell, no cross-cell calls
Poisonous requestone customer's calls break a shared fleetshared worker poolshuffle sharding, per-account limits
Control plane downcannot create, but existing instances runcontrol-plane dependency in the data pathstatic stability: data plane keeps last known state

Real implementations

namewhat it isrepolicencenotes
Amazon EC2 APIthe reference: signed requests, client tokens, token-bucket throttlingEC2 developer guide—closed, documented
OpenStack Novacompute API, scheduler, conductor, cellsopenstack/novaApache-2.0 (LICENSE)architecture
OpenStack Neutronvirtual networks, pluggable IPAMopenstack/neutronApache-2.0 (LICENSE)IPAM
Apache CloudStackopen-source cloud orchestrationapache/cloudstackApache-2.0 (LICENSE)
Azure Resource Managerasync operations with status URLsdocs—closed, documented
Google API guidelineslong-running operations and request idsAIP-151—design rules, not code
Temporaldurable workflow engine usable for provisioning stepstemporalio/temporalMIT (LICENSE)
PhysaliaAWS cell-oriented control-plane store (paper)NSDI 2020—closed, described

The alternative: one synchronous call that does everything inside the request is simple and fine for a few resources that start in under a second; it fails once steps take minutes, a step can fail halfway, or callers retry, which is why every large cloud uses intent plus operations.

Sources

Similar systems

Share the most building blocks with this one.

multi-tenant-saas 19 %container-orchestrator 17 %auto-remediation 16 %llm-workflow-automation 16 %event-router 15 %