gitops

How it is built

Architecture overview

A GitOps system keeps a running environment equal to a description stored in Git (or another versioned store). A software agent that lives next to the environment, not in the CI system, fetches the description, turns it into concrete resource definitions, compares them with what is actually running, and changes the environment until the two match; then it keeps comparing. The release process shrinks to "merge a change", the audit trail is the commit history, and rollback is a revert. The OpenGitOps project states the idea as four principles: the desired state is declarative; it is versioned and immutable; agents pull it automatically; and agents continuously reconcile actual state toward it.

The one idea that makes it work: the cluster pulls, and the reconcile loop is level-triggered. Because the agent runs inside the trust boundary, the CI system never holds cluster credentials; it only commits. Because the loop compares full desired state with full live state on every pass (and on every change event), a missed webhook, a crashed controller or a hand edit does not matter: the next pass converges again. The two reference systems, Argo CD and Flux, differ in shape (one application controller with a UI and API server, versus a toolkit of small controllers composed through custom resources) but implement this loop.

Parts:

• Source of truth — a Git repository or OCI artifact with versioned, immutable revisions — Content-addressed storage, merkle-tree (no page yet), Digital signatures (Ed25519)
• Change detection — poll on an interval, and wake early on a push notification — Webhook delivery, Message authentication codes (HMAC, CMAC, Poly1305, KMAC)
• Renderer — turn templates, overlays or charts into plain manifests for a revision — Content-addressed storage, Process sandboxing
• Differ and planner — compare rendered desired state with live state, decide what to change — declarative-diff-plan (no page yet), Three-way merge, Optimistic concurrency control
• Reconciler — the loop, with retries and drift correction — Desired-state reconciliation, Watch notifications, Exponential backoff with jitter
• Ordering and health — apply in dependency order and wait for readiness — Topological sort, Health checking, staged-rollout (no page yet)
• Secrets and trust — keep secrets out of plain Git, verify what is fetched — Secrets management, Mutual TLS, Tamper-evident audit log
• Controller runtime — run, stop and restart the agent, and keep its own state recoverable — Process supervision, Graceful draining, Primary-backup replication, built on Point-in-time recovery
• Observability and verification — measure sync latency and test the parser and the loop — hdr-histogram (no page yet), Coverage-guided fuzzing, Deterministic simulation testing

 developer ─PR─► Git (main) ◄──────────── CI builds image, commits new tag (no cluster access)
                  │  │ push event (HMAC-signed webhook)
       poll every N │  ▼
                  ▼  receiver ──► "refresh now"
              source fetch ─► artifact/cache keyed by (repo, revision, path)
                  │
                  ▼ render (kustomize / helm / plain YAML)
          desired manifests ──► diff ◄── live state (list + watch, in-cluster cache)
                                  │ out of sync?
                                  ▼
             plan: create / update / prune, ordered by waves and dependsOn
                                  │ apply (server-side), wait for health
                                  ▼
          cluster ◄──────────── status, events ──► notifications, UI, audit

How the parts fit together

Source of truth

Git stores commits as objects named by the hash of their content, with each commit pointing at its parents and a tree: a hash tree (merkle-tree, no page yet) built on Content-addressed storage. That gives the two properties the principles ask for. A revision is immutable: its id changes if any byte changes. And history cannot be rewritten without the new ids being visible. Flux's source-controller makes the artifacts it fetches addressable by their source identifier (sha, version, timestamp) and packages them as tarballs or YAML for in-cluster consumers. Both systems can check commit signatures: Flux's GitRepository has an optional verify field for commit signatures (reporting a PGP key id or an SSH fingerprint), and Argo CD documents GnuPG signature verification. Signing a commit is Digital signatures (Ed25519) when the key is an Ed25519 key.

The contract: the agent reads one revision at a time and treats it as a closed input. It never writes back, except for things like image-tag updates that Flux's image automation controllers commit on purpose.

Change detection

Polling is the baseline and the reason the loop survives lost notifications. Argo CD's documentation says it polls Git, OCI and Helm repositories every three minutes by default; a webhook removes the delay. Its API server accepts push events from several Git providers and from OCI registries and triggers a refresh. Flux's notification-controller has a Receiver resource: it generates a unique webhook path from the receiver name, namespace and a token, verifies each payload with HMAC through the X-Hub-Signature header, and, if the event type matches, triggers reconciliation of the listed resources.

So the webhook is only a hint. It carries no state the agent needs; the agent re-fetches. That is why the design needs no delivery guarantee from the sender (see Webhook delivery for what senders do and what receivers must tolerate) but does need the receiver to check the MAC (Message authentication codes (HMAC, CMAC, Poly1305, KMAC)) and to be sized against abuse: Argo CD's own documentation notes that the endpoint lacks rate limiting and recommends limiting the payload size.

Renderer

Desired state in Git is rarely final YAML. It is a Helm chart with values, a Kustomize overlay, or a plugin's output. Argo CD isolates this step in the repository server, an internal service that keeps a local cache of the Git repository and returns manifests when given the repository URL, revision, application path and tool settings. Because the output is a pure function of those inputs, it can be cached by the same key; Argo's high-availability notes advise limiting concurrent manifest generations (--parallelismlimit) because the tools are forked as child processes and can exhaust memory. Flux splits it differently: source-controller fetches; kustomize-controller builds the overlay in the same run it applies.

Rendering runs code from the repository (a Helm template, a plugin), so it deserves the isolation of Process sandboxing: no cluster credentials in the renderer, a read-only filesystem, a memory limit. Determinism matters: a chart dependency that floats to "latest" makes two passes over the same commit render differently, and the differ will report phantom drift.

Differ and planner

declarative-diff-plan (no page yet) is the comparison: rendered desired objects against live objects. Naive equality fails, because the live object carries fields set by the API server and by other controllers (defaults, status, injected sidecars, a replica count owned by an autoscaler). The standard answer is the idea of Three-way merge: compare the base (what the tool last applied), the desired state and the live state; changes made only by others are left alone, and a field that both sides changed is a conflict. Kubernetes' server-side apply moves this bookkeeping into the API server: it tracks which manager owns which field, returns a conflict when a field owned by another manager would change, and can be forced, transferring ownership. Flux's kustomize-controller runs a server-side apply dry-run at every interval to detect and correct drift. Argo CD offers ignoreDifferences rules (by JSON pointer, JQ path or managed-fields manager) for fields that someone else legitimately owns, such as /spec/replicas under an autoscaler.

Writes that carry a resource version are rejected if the object changed meanwhile, so a stale write fails and the loop re-reads (Optimistic concurrency control).

Reconciler

The loop itself is Desired-state reconciliation: the Kubernetes controller pattern, in which a controller tracks resources whose spec is the desired state and moves the cluster closer to it. To avoid listing everything every time, it holds a local cache of the live objects fed by list-then-watch (Watch notifications): the list returns a resource version, the watch streams changes after it, and a dropped watch resumes from the last version or re-lists. Argo CD's application controller keeps comparing the live state with the target and marks the application OutOfSync; it only corrects it if auto-sync is on. Auto-sync is conservative by default: it does not prune resources that disappeared from Git, and it does not heal live changes, unless prune and selfHeal are enabled. A protection stops a sync that would delete everything when the rendered result is empty. Flux's concepts page makes the same point from the other side: a change made with kubectl edit is promptly reverted unless reconciliation is suspended.

Failures retry with Exponential backoff with jitter-style delays. Argo CD's retry policy is a limit, a base duration, a factor and a maximum duration (for example 5 seconds, doubling, capped at 3 minutes in the documentation example; documentation lists no jitter parameter). Flux separates the periodic interval from a retryInterval used only after failures.

Ordering and health

Resources depend on each other: a custom resource needs its CRD, a workload needs its namespace and config. Argo CD orders a sync by phase (PreSync, Sync, PostSync hooks), then by wave number, then by kind (namespaces first, then other Kubernetes resources, then custom resources); between waves it waits a configurable delay, two seconds by default, to let other controllers react to the spec just applied before health is assessed. Flux expresses it as dependsOn between Kustomizations: one is applied only after the ones it names are Ready, and healthChecks list the workloads whose rollout the controller waits for. Both are Topological sort over a dependency graph, evaluated per pass, with Health checking as the wait condition: a resource is healthy when its status says so, not when the API accepted the write.

staged-rollout (no page yet) sits above this: promoting a change across environments by merging into one directory after another, or across clusters with Argo CD's ApplicationSet progressive syncs. Traffic-level staging (canaries) is a different layer; see the rollout and Automated canary analysis pages.

Secrets and trust

Git must not hold plaintext secrets, so the repository holds an encrypted form and the agent decrypts in the cluster. Flux documents a SOPS integration for this; the generic problem and options are on Secrets management. The agent authenticates to the Git host with a deploy key or token, and the cluster API with its in-cluster service account. Every change is a commit plus a status event, which together are the audit trail (a commit signature gives authorship); Tamper-evident audit log is the pattern when the trail must resist tampering by the operator, not only by outsiders. Argo CD documents mutual TLS for its repository server (Mutual TLS).

Controller runtime

Process supervision keeps the agent running: a parent starts the controller, reaps its exit and restarts it under a delay and burst limit (in Kubernetes the kubelet and a Deployment do this). Restart is cheap because the loop is level-triggered and holds no unrecoverable state. Graceful draining is the other half: on SIGTERM the controller stops taking new reconcile work, finishes the sync in flight up to a deadline and exits, so an upgrade does not stop half-way through a wave. The agent's own state (applications, sync status, caches) lives in the cluster store. That store is a Primary-backup replication setup (one node orders writes, standbys follow), and Point-in-time recovery restores it to an instant before a bad merge was applied. These two blocks protect the cluster store the agent depends on, not its logic; the repository remains the recovery source for desired state.

Observability and verification

hdr-histogram (no page yet) is the bucketed histogram for sync duration and queue latency percentiles: bounded relative error, mergeable across replicas, so alerts on slow syncs rest on real tails. The manifest parser and the differ take untrusted input from the repository, so Coverage-guided fuzzing targets them: mutate manifests, keep inputs that reach new code edges. Deterministic simulation testing targets the loop: controller, fake API server and clock run in one seeded thread with dropped watches and conflicting writes injected, and any failure replays from its seed.

Data and control flow

A change reaches the cluster.

1. A CI job builds an image and commits the new tag (or a developer edits a manifest) and pushes. Writes: one Git commit. CI needs Git credentials only.
2. A push event hits the receiver, which verifies the HMAC and marks the application for refresh; without a webhook, the next poll (Argo CD: three minutes by default) finds the new revision. Time: seconds with a webhook, up to the interval without.
3. The renderer fetches the revision and builds manifests. Reads: Git; writes: a cache entry keyed by repository, revision and path. Time goes to cloning and the template tool.
4. The differ compares desired with live (from the in-memory cache), producing create, update and prune sets and marking the application OutOfSync.
5. With auto-sync on, the reconciler applies in wave or dependsOn order, waits for health between steps, and retries failures with backoff.
6. Status and events are written back to the custom resource; notifications fire; the loop returns to idle and re-checks on the next interval.

Drift correction. Someone edits a deployment by hand. The live-object watch fires; the differ sees live differ from desired in a field that is not ignored; with self-heal (Argo CD) or the next interval (Flux), the agent re-applies the desired value. No human is told unless an alert is wired to the out-of-sync status.

Rollback. git revert creates a new commit with the old content; the flow above runs unchanged, so rollback takes the same path and time as a deploy. Reverting live objects by hand does not last: the next pass reapplies Git.

Scaling

1. Repository and render load. Many applications in one repository cause repeated clones and renders. Levers: webhook-driven refresh with a long poll interval, manifest caching by revision, a render parallelism limit, split repositories or sparse checkout.
2. Controller memory and CPU grow with the number of tracked live objects. Levers: shard clusters across controller replicas (documented for Argo CD), filter watches to managed kinds, raise intervals.
3. API server load from lists, watches and applies. Levers: informer caches, longer intervals.
4. Fleet size. Hundreds of clusters make "one hub controller" a single point of failure and a credential concentration. Levers: an agent per cluster (the Flux model) or a hub per region.
5. Blast radius. One bad merge reaches every environment tracking the branch. Levers: promote across directories or branches with checks, required review, sync windows and progressive syncs.

Failure modes

Real implementations

The alternative: push-based CI/CD: the pipeline runs kubectl apply or a deploy tool against the cluster after tests pass. It is simpler to start and easy to sequence with tests, but it needs cluster credentials in CI, sees the cluster only at deploy time (so drift goes unnoticed), and a failed pipeline can leave the environment half-changed. Choose pull-based GitOps for many clusters, strict credential boundaries and drift control; choose push for a single environment, non-Kubernetes targets with no agent, or when the deployment must be sequenced with external steps that an in-cluster agent cannot see.

Sources

• OpenGitOps, GitOps principles v1.0.0.
• Argo CD docs: architecture, automated sync, sync phases and waves, webhook configuration, diffing, high availability, GnuPG verification.
• Flux docs: concepts, components, source controller, GitRepository, Kustomization, Receiver, SOPS guide.
• Kubernetes docs: Controllers, Server-Side Apply, API concepts (resource versions, watch).