autoscaler

How it is built

Architecture overview

An autoscaler keeps the capacity of a cluster matched to its load, in two layers that run as separate control loops. The workload scaler (Kubernetes' HorizontalPodAutoscaler is the reference) reads a metric such as CPU use or requests per second and changes the number of replicas of a service. The node provisioner (Cluster Autoscaler, Karpenter, EC2 Auto Scaling groups) watches for replicas that cannot be placed because no machine has room, and adds machines; when machines are mostly empty it drains and removes them. The two layers talk only through the scheduler: more replicas create pending pods, pending pods trigger nodes.

The idea that makes it work is feedback control with deliberate slowness. Adding capacity is cheap and urgent, removing it is cheap to postpone and expensive to get wrong, and the signal is noisy and delayed (metrics lag, machines take minutes to boot). So scale-up reacts fast and scale-down waits, using tolerance bands, stabilization windows and cooldowns to avoid oscillation. The cost lever sits in the node layer: choosing which instance types to launch (including interruptible spot machines) and packing pods tightly enough to empty the rest, so that a bin-packing problem and a purchasing problem are solved by the same component.

Parts:

• Signal — usage and request metrics, smoothed and filtered — resource-usage-accounting (no page yet), Sliding-window counters, Exponential smoothing, Robust statistics
• Workload scaler — turns a metric into a replica count — autoscaling (no page yet), PID control
• Forecaster — scales ahead of predictable load — Time-series forecasting, STL decomposition, arima (no page yet), Kalman filter, Time-series cross-validation, croston-method (no page yet)
• Node provisioner — picks and launches instance types for pending pods — Bin packing, spot-capacity-allocation (no page yet), warm-pool-provisioning (no page yet)
• Consolidation and eviction — removes or replaces nodes safely — priority-and-preemption (no page yet), Graceful draining, Health checking, D-ary heap
• Control plane — the shared reconcile loop, API watches and leader election — Desired-state reconciliation, Watch notifications, MVCC, Leases, Fencing tokens, Exponential backoff with jitter, conditional-writes (no page yet)

 pods ──metrics──► signal (window, smooth) ──► WORKLOAD SCALER ──► replicas = N
                                                   ▲                    │
                          forecast (daily/weekly) ─┘                    ▼
                                                            scheduler: place pods
                                                                        │ no room
                                               pending (unschedulable) pods
                                                                        ▼
 cloud API ◄── launch node(s) ◄── NODE PROVISIONER: bin-pack, choose types, spot/on-demand
     │                                  ▲
     └── node ready ──► scheduler       └── CONSOLIDATION: drain + delete underused nodes

How the parts fit together

Signal

The scaler is only as good as its input. Two kinds exist: utilisation (CPU or memory used divided by requested, from the kubelet and a metrics server) and demand (queue depth, requests per second, from an application or message broker). Utilisation is the easy default but lags: a CPU-bound service saturates before the average moves. Raw values are noisy, so they pass through a window (Sliding-window counters for rates) and a smoother (Exponential smoothing), and single bad samples from a crashed or starting pod are damped with robust statistics (a median or trimmed mean instead of the mean). Per-workload usage numbers come from cgroup counters sampled over time (resource-usage-accounting, no page yet).

Pods that are still starting report odd CPU. Kubernetes' HPA therefore treats pods that are not ready, or that have just started, specially when computing the average (source: Kubernetes docs, HPA).

Workload scaler

The HPA rule is one line: desiredReplicas = ceil(currentReplicas × currentMetricValue / desiredMetricValue) (source: Kubernetes docs, HPA). If the metric is twice the target, double the replicas. It is a proportional controller (PID control without the integral and derivative terms), with two additions that matter more than the formula:

• a tolerance (default 0.1): no action if the ratio is within 10 % of 1.0, a dead band that stops chattering (source: same docs);
• a stabilization window for scale-down: the controller keeps recent recommendations and acts on the highest within the window, 5 minutes by default (source: same docs), so a brief lull does not remove capacity just before the next peak.

The generic technique, autoscaling (no page yet), is target tracking with cooldowns, hysteresis and rate limits on how fast the count may change. Scaling on several metrics takes the largest result. The scaler writes only the replica count on the target object; everything else happens through the object model.

Forecaster

Reactive scaling is always late by the boot time of the new capacity. If load follows daily and weekly rhythms, a forecast lets the system add capacity before the peak. EC2 Auto Scaling's predictive scaling does this by analysing historical load to find daily or weekly patterns and forecast capacity needs (source: AWS docs, predictive scaling). The standard recipe is: split off the seasonal pattern (STL decomposition), forecast the remainder with a baseline such as Exponential smoothing or arima (no page yet), and validate with rolling-origin backtests (Time-series cross-validation), never on random splits. A Kalman filter can track a noisy, drifting level in real time. Intermittent demand (a job that runs a few times a day) fits croston-method (no page yet), not a smooth forecast.

Forecast with a safety margin and keep the reactive loop on: the forecast sets a floor, the live signal can still raise it.

Node provisioner

When pods are pending, someone must choose machines. Cluster Autoscaler works on node groups with fixed machine shapes: it simulates scheduling the pending pods onto a template node of each group and expands a chosen group; its stated goal is a latency of at most about 20 seconds in its scalability tests from "pod marked unschedulable" to "scale-up request sent" (source: Cluster Autoscaler FAQ). Karpenter works without fixed groups: it watches unschedulable pods, reads their requests and scheduling constraints, and launches nodes sized to fit them from a flexible list of instance types constrained by a NodePool, which also sets limits and weights (source: Karpenter docs, concepts and NodePools).

The core of that decision is Bin packing: fit the pending pods, each with a CPU and memory size, into as few and as cheap machines as possible. It is NP-hard in general, so first-fit-decreasing style heuristics are used: sort pods by size, place each into the first machine that fits, open a new machine type only when none does. Spot capacity adds a second dimension, spot-capacity-allocation (no page yet): spread over many instance types and zones, because each pool has its own price and interruption rate, and do not always pick the single cheapest. Karpenter enforces a minimum flexibility of 15 instance types for single-node spot-to-spot consolidation for that reason (source: Karpenter docs, disruption).

The slowest part of scale-up is the machine itself: boot, join the cluster, pull images. Warm pools (warm-pool-provisioning, no page yet) keep pre-initialised stopped or running instances ready to cut that delay (source: AWS docs, warm pools), traded against a small standing cost. Health checking decides when a new node counts as ready; a node that never becomes ready must be replaced, not waited for.

Consolidation and eviction

Scaling down is harder than scaling up because it moves running work. Karpenter's disruption methods are consolidation (remove or replace nodes that cost more than needed), drift (replace nodes that no longer match their NodePool), expiration and interruption handling, run one at a time and throttled by disruption budgets (source: Karpenter docs, disruption). Cluster Autoscaler removes nodes that stay underused for a configured time and whose pods can move elsewhere.

Safe removal needs graceful draining: cordon the node so nothing new lands, evict pods respecting their disruption budgets and termination grace periods, wait, then delete the machine. Which pods may be evicted, and which pending pods may push out others, is the subject of priority-and-preemption (no page yet): a priority class gives an ordering, a cutoff stops low-priority "filler" pods from triggering scale-up. Candidate nodes for removal are held in a priority queue ordered by cost or utilisation (D-ary heap). For spot capacity the provider sends an interruption notice two minutes before reclaiming an instance (source: AWS docs, spot interruption notices); the provisioner uses that window to drain.

Control plane

All of this runs as controllers on desired-state reconciliation: each loop reads the full current state, computes what should change and applies it idempotently, so a missed event only delays convergence. Controllers learn about changes through watch notifications over the API server's MVCC-versioned store, and write with version checks (conditional-writes, no page yet) so two writers cannot overwrite each other. Controller failures are retried with backoff and jitter. Only one replica of a controller may act at a time, which uses a lease held in the API server; a paused ex-leader must not act on stale decisions, which is the problem fencing tokens address, and which the version check on the write also solves.

Data and control flow

A traffic spike (scale up):

1. Traffic rises; CPU per pod climbs. The metrics pipeline reports it with a delay of tens of seconds.
2. On its next sync the HPA computes the ratio, finds it outside the tolerance and raises the replica count on the target.
3. The deployment creates pods; the scheduler cannot place some of them and marks them unschedulable.
4. The provisioner sees the pending pods, batches the ones that arrived together, bin-packs them onto candidate instance types and calls the cloud API to launch machines (spot first if allowed, with fallbacks).
5. The nodes boot and join; the scheduler places the pods; readiness probes pass; traffic is served. Time goes into steps 1 (metrics), 4 (API) and 5 (boot and image pull); the cloud boot dominates unless a warm pool is used.

Load drops (scale down):

1. The metric falls below target. The HPA records lower recommendations but acts on the highest in its window, so replicas shrink only when the whole window agrees.
2. Pods terminate; nodes become underused.
3. The provisioner finds nodes that can be removed because their pods fit elsewhere or a cheaper single node can replace several, within the disruption budget.
4. It cordons, drains and deletes the node.

Predictive pre-scaling:

1. Each day the forecaster fits the history with a seasonal model and writes a capacity schedule for the next hours.
2. Capacity is raised ahead of the predicted peak by the lead time of a machine.
3. The reactive loop keeps running and may raise capacity further.

Scaling

What limits the autoscaler itself, in the order you meet it:

1. Reaction time is metric lag plus machine boot time. Levers: faster metrics (push-based, demand metrics such as queue length), smaller images and faster boot, warm pools or a small spare buffer of low-priority placeholder pods that are evicted when real pods arrive, forecasting.
2. Cloud API limits and capacity. Large scale-ups hit rate limits and "no capacity for this instance type" errors. Levers: flexible instance type lists, several zones, batching launches into one call, backoff with jitter.
3. Packing quality sets cost. Levers: correct pod resource requests (the scaler can only pack what is declared), larger machines for many small pods, consolidation, and moving to flexible provisioning instead of fixed node groups.
4. Cluster size: each decision considers all pending pods and candidate nodes, so decision time grows with cluster and pod count. Levers: batch windows, per-NodePool limits, splitting workloads across clusters.
5. Controller availability: one active replica by lease; failover takes a lease duration. Levers: shorter leases within reason, idempotent loops.

Failure modes

Real implementations

Choice: HPA plus KEDA for the workload layer; Cluster Autoscaler when node groups are fixed and well understood; Karpenter when instance flexibility, spot use and fast consolidation are the main savings.

Sources

• Kubernetes docs, HorizontalPodAutoscaler: ratio formula, tolerance of 0.1, downscale stabilization of 5 minutes.
• Karpenter docs: overview, NodePools, scheduling, disruption (consolidation, drift, budgets, the 15-instance-type spot-to-spot minimum).
• Cluster Autoscaler FAQ: scale-up latency goal, priority cutoff.
• AWS docs: predictive scaling, warm pools, Spot interruption notices (two-minute warning).
• KEDA, documentation.
• Licence fields read through the GitHub API.