ci-server

How it is built

Architecture overview

A CI server turns "someone pushed a commit" into "a graph of jobs ran on isolated machines and the result is on the pull request". It has one durable brain and many disposable hands. The brain is the controller: it receives events from the code host, loads the pipeline definition, expands it into a graph of jobs, keeps a durable queue of jobs that are ready, and records every state change. The hands are agents (GitHub calls them runners): processes on other machines that ask the controller for work, run it under isolation, stream logs back and upload artifacts. Around that core sit a secrets and identity layer (the job needs credentials but must not be able to steal them), an artifact and cache store, and a state store that has to survive a controller crash. The one idea that makes it work: agents pull, the controller only records. An agent that vanishes simply stops renewing a lease, the job returns to the queue, and the controller never needs to reach into a build network. Everything else (retries, priorities, skipping unchanged work) is bookkeeping on top of that loop.

Buildkite's agent polls Buildkite over HTTPS, so no inbound firewall rule is needed (source: Buildkite docs); the GitHub Actions runner holds a long-poll connection to the Actions service (source: ARC docs); Jenkins agents connect to the controller (source: Jenkins docs).

Parts:

• Event intake — receive and authenticate code-host events, acknowledge fast, never lose one — Webhook delivery, Message authentication codes (HMAC, CMAC, Poly1305, KMAC), hmac-request-signing (no page yet), Idempotency keys
• Pipeline planner — turn the definition into a job graph and decide what can be skipped — DAG workflow execution, glob-matching (no page yet), Content-addressed storage
• Job queue and scheduler — hold ready jobs, hand each to exactly one agent at a time, retry — Durable job queue, d-ary heap, Exponential backoff with jitter, Leases
• Agent channel — how agents ask for work and how logs and status come back — Long polling, Server-sent events, Mutual TLS
• Executor sandbox — run untrusted job steps and kill them cleanly — Process supervision, Least privilege and isolation
• Identity and secrets — give a job short-lived credentials, not stored keys — Workload identity (SPIFFE-style), Secrets management
• Artifacts and cache — pass outputs between jobs and runs — Content-addressed storage, Object storage
• State store and availability — keep run history, survive and replace the controller — Primary-backup replication, Point-in-time recovery, Graceful draining
• Verification and observability — prove the scheduler is right, see queue wait — Deterministic simulation testing, Coverage-guided fuzzing, hdr-histogram (no page yet)

 code host ──push/PR event──► event intake ──► planner ──► job queue ◄──poll/long-poll── agents
 (signed webhook)             verify, dedupe    graph,      (leased jobs,   (lease+heartbeat)  │
        ▲                          │            skip        priority heap)        │            ▼
        │                          ▼            unchanged        │              ▼        executor sandbox
        │                    state store ◄──────────────────────┘          log + status    (process tree,
        │                  (runs, jobs, logs idx)                           stream ──► SSE     isolation)
        │                          │                                         to UI               │
        └── status checks ◄────────┴──► artifact + cache store (object storage, by content hash) ◄┘
                                       secrets / job identity issuer ──► short-lived token to the job

How the parts fit together

Event intake

The code host tells the CI server about a push or pull request by sending a webhook. Webhook delivery describes the sending side; the CI server is the receiver, so its contract is: verify the signature on the raw request bytes, store the event, answer 2xx, and do all real work later. GitHub signs deliveries with an HMAC-SHA256 hex digest in a header and asks receivers to compare it in constant time (source: GitHub docs, "Validating webhook deliveries"); Message authentication codes (HMAC, CMAC, Poly1305, KMAC) is the primitive, and hmac-request-signing (no page yet) the request-level recipe. GitHub expects a 2xx within 10 seconds and does not redeliver failed deliveries on its own (source: GitHub docs, "Handling webhook deliveries" and "Handling failed webhook deliveries"). That sets the intake's two rules. It must be tiny (verify, write one row, answer) so it never misses the deadline, and it must be idempotent with the delivery id as key (Idempotency keys), because redelivery by hand or by script produces duplicates. Because a missed event is lost for good, a robust server also polls or reconciles the code host periodically; the webhook is a hint, the repository is the truth.

Pipeline planner

The definition file (YAML, Groovy, Starlark) becomes a graph: jobs, needs edges, matrix expansions, filters. DAG workflow execution is the engine: reject cycles at submission, count unfinished inputs per job, release a job at zero, prefer the job with the longest chain behind it. Branch and path filters (on: push: paths: ["src/**"]) are glob matches against changed files; a naive regex translation can blow up, so glob-matching (no page yet) is the right block. The same engine gives "skip what did not change": a job's identity is a hash of its inputs and its dependencies' hashes, stored in Content-addressed storage. A build-system-style planner that skips by input hash is the stronger form of this; see the build-system page of this series. The contract between planner and queue is small: the planner emits job specs (id, run id, attempt, required labels, timeout, input digest) and never talks to agents.

Job queue and scheduler

A Durable job queue holds jobs that are ready. An agent claims a job under a Leases term, renews it with heartbeats while the job runs, and acks at the end; if the heartbeat stops, the lease expires and the job is requeued. This is the whole answer to "what if the agent machine dies". A paused agent may report after the job was reassigned, so result writes carry a fencing token and stale ones are rejected (Fencing tokens). The ready set is ordered by a priority key (queue age, branch priority, pipeline); a d-ary heap is the cheap structure for it. A failed job retries with Exponential backoff with jitter when the failure looks infrastructural (agent lost, image pull failed) and goes to a Dead-letter queue after a retry limit; a test that fails is a result, not a retry. Label matching (runs-on, queue, tags) is a filter at claim time: the agent presents its labels and gets the oldest compatible job. Buildkite additionally prefers agents that most recently finished a job, to benefit from warm caches (source: Buildkite docs). Contract: at-least-once execution, so a job step that deploys or publishes must itself be idempotent.

Agent channel

Agents pull. Long polling fits: the agent asks "any job for my labels?", the controller holds the request until there is one or a timeout, and the agent asks again immediately. This works through NAT and firewalls with outbound HTTPS only. Woodpecker agents ask the server for work over gRPC (source: Woodpecker docs, agent configuration), and the Gitea runner polls a Gitea instance for queued jobs (source: Gitea docs). The reverse direction (controller connects to agent) needs inbound access to every build network. The agent authenticates with a registration token and then a per-agent credential; where agents run on infrastructure you control, Mutual TLS gives each one a rotatable machine identity.

The log path is the opposite shape: many bytes, one direction, browser at the end. The agent uploads log chunks over its own channel; the UI tails them with Server-sent events, where each event carries an id so a reconnecting browser resumes from the last line it saw. Contract: chunks are append-only and numbered, so the controller can store them as a segment list and the UI can fetch "everything after chunk N".

Executor sandbox

An agent runs job steps as child processes, in a container, or in a throwaway VM. Process supervision is what the agent itself needs: start the step, reap it, enforce the timeout, send SIGTERM then SIGKILL to the whole process tree so a stuck test does not hold the slot. Least privilege and isolation is the trust boundary. Pull-request builds run code written by someone the project may not trust, so the job gets no network route to the controller's internals and no credentials beyond its own. Jenkins documents the consequence of getting this wrong: an agent taken over by an attacker could still obtain data or run commands on the controller unless agent-to-controller access control is enforced (source: Jenkins docs, "Controller isolation"), and it recommends not building on the built-in node. How a fleet of such sandboxes is provisioned and scaled is a separate system, the CI runner fleet page of this series.

Identity and secrets

A job needs to push an image, deploy, read a package: it needs credentials. Static keys stored in the CI server are the classic breach. Workload identity (SPIFFE-style) replaces them: the controller (or the agent) attests which job this is (repository, branch, workflow), then issues a short-lived signed token that the target system accepts through trust configuration. No secret is stored; a leaked token expires in minutes. For what cannot be federated (a third-party API key), Secrets management covers encrypted storage, injection only into the steps that need them, and masking in logs. Contract: no secrets for jobs triggered from untrusted forks; masking is best-effort, not a boundary.

Artifacts and cache

Jobs pass files to later jobs (build output to test, test report to publisher) and between runs (dependency caches). Both go to Object storage, keyed by Content-addressed storage hash for caches and by run id for artifacts, because the controller's disk is the wrong place for gigabytes: Jenkins notes that files written during a pipeline land on the controller's filesystem unless offloaded to an artifact repository (source: Jenkins docs, "Managing Nodes"). Content addressing makes upload idempotent and lets an agent verify a download.

State store and availability

The controller's state (runs, jobs, leases, log index, audit trail) must survive a crash and an upgrade. Jenkins documents the cost of its durability choice: pipelines write transient data to disk frequently so running pipelines survive a restart, and a faster setting trades part of that away (source: Jenkins docs, "Scaling Pipelines"). Two layers sit under every design. Primary-backup replication keeps a warm standby, and the acknowledgement rule decides whether a crash can lose the last job state change. Point-in-time recovery rebuilds the state at an instant before a corrupting upgrade or a bad migration. Concourse runs several web nodes against one shared database (source: Concourse docs, internals). For rolling the controller itself, Graceful draining stops it from taking new webhook and agent requests, lets in-flight claims finish or hand off, and tells agents to reconnect to the next instance.

Verification and observability

Scheduler bugs (two agents run one job, a job never leaves the queue after a lease race) show up only under failures that are hard to reproduce. Deterministic simulation testing runs controller, queue and simulated agents under one seeded scheduler with injected crashes, delays and clock jumps, so a failing seed replays exactly. The parts that parse input from outside (pipeline YAML, webhook JSON, log chunk framing) get Coverage-guided fuzzing. The metric users feel is queue wait at a high percentile; hdr-histogram (no page yet) measures it with bounded error.

Data and control flow

One push, end to end

1. The code host sends a signed webhook. Intake verifies the HMAC on the raw body, checks the delivery id against the dedupe table, writes the event, answers 2xx. Budget: well under the host's 10 s timeout.
2. The planner loads the pipeline file at that commit, applies branch and path filters, expands matrices, builds the DAG, and computes each job's input hash. Jobs whose hash matches a prior successful run are marked cached.
3. Jobs with no unfinished inputs are inserted in the durable queue; state changes go to the state store in the same transaction as the queue insert.
4. An agent's long poll returns a job: the queue grants a lease with a fencing token and returns the job spec plus a short-lived job token.
5. The agent starts the job in its sandbox, heartbeats the lease, uploads log chunks, and streams them to the UI via SSE.
6. On finish the agent uploads artifacts to object storage and acks with the fencing token. The planner decrements the dependents' counters and releases the next jobs. When the last job ends, the server posts the result to the code host as a status check.

One lost agent

1. The agent's machine is preempted; heartbeats stop.
2. The lease expires; the queue returns the job with attempt+1 after a backoff with jitter.
3. Another agent claims it. If the first agent comes back and tries to ack, the stale fencing token is rejected.
4. After the retry limit, the job goes to the dead-letter queue and the run is marked failed with the reason "infrastructure".

Scaling

In the order they bite:

1. Queue wait is the first user-visible limit: more agents, then autoscaling them (the CI runner fleet page). The controller is rarely the bottleneck for job count; the agents are.
2. Controller write rate (log chunks, heartbeats, state changes): batch heartbeats, move log bytes straight to object storage and keep only an index in the state store, then shard the queue by tenant or label set.
3. Single-controller limit: Jenkins describes both bigger controllers and more controllers; the second needs the state to be partitioned by team or project (source: Jenkins docs, "Architecting for scale"). Stateless web nodes over one database (Concourse) move the limit to the database.
4. Webhook burst: intake only enqueues, so the planner can work at its own pace.
5. Artifact bandwidth: keep object storage near the agents and dedupe by content hash.

Failure modes

Real implementations

The alternative: a push model, where the controller dials each build host. It removes the poll loop, but the controller must reach every host. A Kubernetes-native design (Tekton) uses neither: jobs are pods and the API server is the state store.

Sources

• Jenkins docs: Managing Nodes, Scaling Pipelines, Controller isolation, Architecting for scale.
• GitHub docs: Self-hosted runner reference, Workflow syntax (concurrency groups), Validating webhook deliveries, Handling webhook deliveries, Handling failed webhook deliveries.
• Actions Runner Controller README (runner long-poll connection): docs/gha-runner-scale-set-controller.
• Buildkite: Agent overview. Woodpecker: agent configuration. Concourse: internals. Gitea: Gitea Runner. Tekton: overview. GitLab: Runner autoscaling.