pki

How it is built

Architecture overview

A certificate authority (CA) vouches for the link between a name and a public key. A server proves, once, that it controls a domain name; the CA signs a certificate that says "this key belongs to this name, from this date to that date"; every browser or client that trusts the CA can then check the signature, instead of having to know the server beforehand. Public web security rests on this: a TLS 1.3 handshake ends with the client verifying a certificate chain up to a root it already trusts.

For many years, getting a certificate was a manual job: generate a key, make a request, email a document, wait, install the result, and remember to repeat it every year or two. ACME (RFC 8555) turns the whole job into a protocol between a client program and the CA. The client creates an account, orders a certificate for some names, proves control of each name by answering a challenge, submits a certificate signing request, and downloads the result. Because nothing in it needs a human, lifetimes can be short and renewals frequent, which in turn limits the damage of a stolen key. Let's Encrypt, which runs on the open-source Boulder software, describes the CA's job as automatically verifying that an applicant controls an identifier, and letting subscribers issue and revoke certificates for the identifiers they control (source: Boulder README).

So the system has two faces. The public face is a web API speaking ACME, plus the places where the world checks what the CA did: revocation lists and certificate transparency logs. The private face is a small, tightly guarded core that holds signing keys and decides what may be signed. Almost every design choice in the system is about keeping those two apart.

Parts:

• ACME front end — the HTTP API: accounts, orders, challenges, finalize, revoke; every request signed and replay-protected — acme-protocol (no page yet), built on JSON Web Token (JWT), HTTP/1.1 message parsing, ECDSA signatures, x509-certificates (no page yet), Coverage-guided fuzzing
• Domain validation — checks that the applicant controls the name, by an HTTP file or a DNS record — DNS message format and client lookups, HTTP/1.1 message parsing
• Issuance and certificate format — builds and signs the certificate, with names, key usage and validity — x509-certificates (no page yet), ECDSA signatures, rsa-cryptosystem (no page yet), Cryptographic hashes (SHA-2, BLAKE3), csprng (no page yet), Deterministic simulation testing
• Key protection — keeps CA private keys in hardware and controls who may use them — Hardware security module (HSM)
• Revocation — tells clients which certificates must no longer be trusted — certificate-revocation (no page yet), built on Bloom filter, HTTP caching semantics, ECDSA signatures
• Transparency — publishes every issued certificate to public append-only logs and embeds the proof — signed-certificate-timestamps (no page yet), Tamper-evident audit log
• Abuse limits — caps requests and failures per account and per name — Token bucket
• Transport and trust consumers — the TLS handshake in which clients use the certificates — TLS 1.3, Mutual TLS
• Operations — keeps the CA running, fast and recoverable — hdr-histogram (no page yet), Process supervision, Graceful draining, Primary-backup replication, Point-in-time recovery

 ACME client (certbot, ...)                                         browsers / TLS clients
   │  JWS-signed POSTs over HTTPS                                          ▲  verify chain, SCTs,
   ▼                                                                       │  revocation
 ┌──────────┐   ┌────────────────┐  validate: HTTP-01 / DNS-01  ┌──────────────────┐
 │ WFE      │──►│ Registration   │─────────────────────────────►│ Validation       │──► client's web / DNS
 │ (public) │   │ Authority (RA) │◄──────── result ─────────────│ Authority (VA)   │
 └────┬─────┘   └───────┬────────┘                              └──────────────────┘
      │ rate limits     │ "sign this precertificate"
      ▼                 ▼
 ┌──────────┐   ┌────────────────┐  precert  ┌──────────────────┐  SCTs   ┌────────────────┐
 │ Storage  │◄──│ CA (HSM keys,  │──────────►│ Publisher        │────────►│ CT logs        │
 │ Authority│   │ policy checks) │◄──────────│ (to CT logs)     │◄────────│ (Merkle trees) │
 │ (DB)     │   └────────────────┘   SCTs    └──────────────────┘         └────────────────┘
 └──────────┘           │ final certificate
                        ▼                                  CRL updater ──► CRLs in object storage
                  back to client                                              (fetched by clients)

How the parts fit together

ACME front end

ACME's protocol is HTTPS carrying JSON, with each request wrapped in a JSON Web Signature (JWS) made with the account key (source: RFC 8555). The CA publishes a directory of URLs: new nonce, new account, new order, revoke certificate, key change (source: RFC 8555). Two ideas keep it safe. The request is signed, so the CA knows which account is speaking and a network observer cannot alter it; the signature format is the one covered in JSON Web Token (JWT) (JWS uses the same machinery). And each request carries a nonce that the server issued earlier in a Replay-Nonce header; a nonce is accepted once, so a captured request cannot be replayed (source: RFC 8555).

Boulder's web front end is the component that ACME clients talk to. Internally, the logic is built around five kinds of objects that mirror ACME: accounts, authorizations, challenges, orders and certificates (source: Boulder README). A client's life cycle is: create an account; create an order listing names; for each name, complete one challenge so that the authorization becomes valid; finalize the order by sending a certificate signing request; download the certificate. The front end parses HTTP carefully (HTTP/1.1 message parsing): body limits, content types, and strict JSON handling are part of the security surface, because it is exposed to everyone.

The front end parses signed JSON bodies, headers and certificate signing requests from strangers. Coverage-guided fuzzing mutates request bodies, signature headers and CSR bytes against the instrumented parsers, so a crash or a bad accept in the parser is found in testing, before it can become a wrongly issued certificate.

Domain validation

Before signing, the CA checks the applicant controls the name. Let's Encrypt documents two main challenge types:

• HTTP-01: the client publishes a token at a well-known path on the domain; the CA fetches it over HTTP on port 80, following redirects (up to 10 in Let's Encrypt's implementation). It cannot be used for wildcard certificates (source: Let's Encrypt challenge types).
• DNS-01: the client publishes a TXT record under the name. It is harder to automate than HTTP-01 but works when the web server is unreachable and is the way to get wildcard certificates; CNAME or NS delegation lets the record live in a different zone than the main one (source: Let's Encrypt challenge types).

This is where DNS message format and client lookups and HTTP/1.1 message parsing are used by the CA itself, as a client. The validation component is the part most exposed to hostile networks, because it makes outbound requests to addresses chosen by applicants. Its design principles: run from a segregated network, refuse to connect to non-public addresses, bound response sizes and time, and (in mature CAs) validate from several vantage points so that one hijacked network path cannot fool the CA. A challenge proves control only at the time of the check.

Boulder separates this on purpose: the Validation Authority needs internet access, which puts it at greater risk of compromise, while the Registration Authority can live without internet connectivity and the Certificate Authority needs only to receive instructions from the Registration Authority (source: Boulder README).

Issuance and certificate format

Issuance turns an approved request into an X.509 certificate (x509-certificates, no page yet; RFC 5280). A certificate holds the subject public key, the names (subject alternative names), the validity period, key usage and extension fields, the issuer, a serial number, and the issuer's signature. Three design rules matter:

• The CA decides the content; the applicant does not. The certificate signing request supplies the public key and requested names, but the CA builds every other field from its own policy. A request cannot smuggle in extensions or longer validity.
• Serial numbers are unpredictable. They come from a cryptographically secure random generator (csprng, no page yet) so that an attacker cannot prepare a colliding request.
• Linting before signing. A policy check, and in good implementations a certificate linter, runs on the to-be-signed certificate before the signing key sees it. Boulder's source tree includes a linter and a policy package, and uses a precertificate step (source: Boulder repository).

The signature algorithm is ECDSA signatures or rsa-cryptosystem (no page yet) over a hash from Cryptographic hashes (SHA-2, BLAKE3). The work is dominated by validation and the checks around signing, not by the cryptography.

Deterministic simulation testing runs the front end, validation, issuance and database in one thread with a seeded clock, network and disk, including crashes between the database write and the signing step. The checks are that an order never produces two certificates, no certificate is issued without a passed validation, and the simulated expiry and renewal clock behave; a failing run replays from its seed.

Key protection

A CA private key is the most valuable secret in the system: whoever has it can issue for any name. The design is layered. A root key is kept offline in an HSM, used rarely (to sign intermediates) under dual control and a ceremony. Intermediate keys do the day-to-day signing and live in online HSMs, so the root stays out of reach. Boulder's tree includes a pkcs11helpers package for talking to HSMs (source: Boulder repository). The HSM enforces that the key cannot be extracted, and the signing service enforces what may be signed: the CA component receives instructions only from the registration authority (source: Boulder README) and applies its own checks.

If an intermediate is compromised, the root revokes it and issues a replacement; if the root is compromised, every client's trust store must change, which is why the root's protection is highest.

Revocation

Sometimes a certificate must stop being trusted before it expires: its private key leaked, or it was issued in error. The first tool is a certificate revocation list (CRL), a signed list of revoked serial numbers published by the CA, described in RFC 5280. Let's Encrypt publishes revocation information as CRLs, and says some browsers check them (source: Let's Encrypt revoking page). An online status protocol (OCSP, RFC 6960) answers per certificate with good, revoked or unknown. Large CRLs are expensive to ship to every client, so compact approaches distribute filters that summarise the revoked set, such as Bloom filter based ones; their false positives need a secondary check. CRLs are static documents served from object storage and CDNs, so HTTP caching semantics decide how fresh a client's copy is.

Revocation in ACME is itself an authenticated request. The requester must prove authority in one of three ways: from the account that issued the certificate, from a different authorised account, or by signing with the certificate's private key (source: Let's Encrypt revoking page). Reason codes follow RFC 5280; for example "keyCompromise" when the key is believed exposed (source: same page). The proof-by-key route has a useful property: whoever holds the leaked key can revoke the certificate, so a finder can revoke it even without the owner's account.

Short lifetimes are the other half of the answer: a certificate that lives only weeks limits how long a missed revocation matters. Let's Encrypt's integration guide speaks of its current 90-day certificates and of validity periods under 10 days.

Transparency

Certificate Transparency (CT, RFC 6962) makes misissuance visible: every certificate is recorded in public, append-only logs, which are Merkle hash trees that allow anyone to check that a log only ever grew and that a given certificate is in it (source: RFC 6962). The CA submits a precertificate to several logs, receives a signed certificate timestamp (SCT) from each, a signed promise to include it, and embeds the SCTs in the final certificate. Browsers can then refuse certificates that lack proof of logging.

Let's Encrypt states that it submits all certificates it issues to CT logs and also runs its own, and that those logs are split by expiration date into temporal shards so each shard has bounded size (source: Let's Encrypt CT logs page). The Tamper-evident audit log block is the general form of what a CT log is: a Merkle structure with signed tree heads. The CA keeps its own internal audit log of issuance decisions in the same style.

Abuse limits

A public issuance API will be hammered by honest clients with bugs and by attackers. Limits keep it usable. Let's Encrypt computes its rate limits with a token bucket: bursts up to the full limit are allowed, and capacity refills over time (source: Let's Encrypt rate limits). Examples from the documentation: up to 300 new orders per account every 3 hours, refilling at 1 order every 36 seconds; up to 5 authorization failures per identifier per account per hour, refilling at 1 every 12 minutes, enforced by blocking new orders for that identifier until capacity returns (source: same page). A certificate may include up to 100 identifiers depending on its profile (source: same page); the documentation distinguishes renewals of an identical set of names from new issuance for some limits.

Transport and trust consumers

Certificates exist to be used. In TLS 1.3, the server sends its certificate chain and proves it holds the private key; the client verifies each signature up to a trusted root, checks that the names match, the dates are valid, and (depending on policy) revocation and SCT evidence. The CA's own API is also served over TLS. A TLS server must reload a renewed certificate without dropping connections.

Not every client is a browser. Between the CA's own components (front end, validation workers, the signing core, the log submitters) Mutual TLS gives each component a certificate identity, so only the front end can ask the signing core for a signature and only an approved validator can mark an authorization as passed.

Operations

Process supervision keeps the validation workers, the issuance service, the revocation publisher and the log-submission workers running and restarts them when they exit. Graceful draining is how the public ACME front end is rolled: it stops taking new orders, lets in-flight finalize requests complete, and tells clients to retry elsewhere, so a deploy does not leave an order half-issued. Primary-backup replication keeps a standby of the CA database (accounts, orders, authorizations, issued-certificate records), where the primary orders every write and the standby takes over only when an external arbiter says so. Point-in-time recovery lets operators rebuild that database as of a chosen instant after a bad migration or mistaken bulk revocation; the issued-certificate records it restores are what revocation lists and transparency submissions are rebuilt from. hdr-histogram (no page yet) records latency of the paths clients wait on (new order, challenge validation, finalize, revocation status fetch) as percentiles.

Data and control flow

Issuing a certificate:

1. The ACME client fetches the directory and a nonce, then creates an account with a signed request. The account key identifies it from then on.
2. It submits a new order for one or more names. Rate limits are checked (orders per account, certificates per name set).
3. For each name the CA returns an authorization with challenges. The client provisions the HTTP-01 file or DNS-01 TXT record and tells the CA to check.
4. The validation component performs the check from the CA side, records success or failure, and updates the authorization. Repeated failures count against the failure limit.
5. When all authorizations are valid, the client generates a key pair, sends a certificate signing request to the finalize URL.
6. The registration component checks the request against policy, passes a signing instruction to the signing component, which builds a precertificate, lints it, and signs it with the HSM-held key.
7. The precertificate goes to several CT logs; the SCTs come back; the final certificate with embedded SCTs is built, signed, and stored with the order.
8. The client downloads the certificate and installs it. All steps are written to the audit log.

Renewing: Let's Encrypt recommends checking ACME Renewal Information (ARI) for each certificate at least twice a day and renewing when the CA suggests; as a backstop, renew when a third of the lifetime is left, which for 90-day certificates means 30 days before expiry; for certificates valid under 10 days, renew halfway; and for more than 10,000 hostnames, renew in small runs rather than one large batch (source: Let's Encrypt integration guide).

Revoking: a request signed by the issuing account, another authorised account, or the certificate's key; the CA records the revocation, includes it in the next CRL, and answers status queries accordingly.

Scaling

1. Request volume is dominated by cheap operations (nonces, directory, order status) that scale horizontally behind a load balancer. Cache what can be cached; keep the signing path narrow.
2. Validation costs outbound network I/O per challenge. Scale it by adding validation workers; protect it with limits; run it in a segregated network.
3. Signing needs HSM capacity. Use several HSMs and several intermediate keys; keep the number of online keys small, since each is a risk.
4. Storage holds accounts, orders, authorizations and certificate records; Boulder's storage authority fronts a relational database (MariaDB in its diagram) and all components talk to it (source: Boulder README). Prune expired authorizations and orders; keep issued-certificate records for as long as policy requires.
5. Revocation publishing produces large static files; shard CRLs by serial range and serve them from object storage behind a CDN. Boulder's CRL updater writes to object storage that clients fetch from (source: Boulder README).
6. CT logging adds a dependency on external logs. Submit to several in parallel and tolerate slow ones; temporal sharding of logs keeps each log's size bounded (source: Let's Encrypt CT logs page).

Failure modes

Real implementations

Choosing: use a public ACME CA for anything browsers must trust. Use a private CA with an ACME server (such as step-ca) for internal services, where you control the trust store and can use very short lifetimes and your own policy.

Sources

• IETF: RFC 8555 (ACME), RFC 5280 (X.509 certificates and CRLs), RFC 6960 (OCSP), RFC 6962 (Certificate Transparency), RFC 8446 (TLS 1.3).
• Boulder, README and repository: component model, security separation, object types, storage and CRL flow.
• Let's Encrypt documentation: challenge types, rate limits, revoking certificates, CT logs, integration guide.
• smallstep/certificates repository page (description and licence).