customer-data-platform

How it is built

Architecture overview

A customer data platform (CDP) collects events about customers from many places (websites, mobile apps, servers, SaaS tools), cleans them against an agreed shape, ties together the events that belong to the same person, and delivers them to many destinations: a warehouse, an ad platform, an email tool. Its value is the fan-in and fan-out: instrument once, send everywhere. Segment popularised the model; RudderStack, Snowplow and Jitsu are open-source relatives.

Technically a CDP is a reliable event pipeline with an identity layer. The pipeline half has one hard promise: an event that was acknowledged to the sender must reach every destination at least once, in an order that the destination can tolerate, even if the destination is down for hours. The identity half has the opposite character: it is a data-modelling problem (who is this person?) that has no perfect answer and must stay correctable. The RudderStack documentation shows the pipeline half in detail: a gateway that accepts events and acknowledges them only after storing them, a processor that transforms them, and a router that delivers them to destinations, with a database holding events until delivery succeeds (source: RudderStack architecture docs).

Parts:

• Collection endpoint — accepts events from SDKs and servers, authenticates the source, validates the shape — JSON Schema validation
• Durable event log — makes acknowledged events survive crashes and lets each consumer read at its own pace — Segmented log, Consumer offsets
• Deduplication — drops events that were sent twice — Idempotency keys, Bloom filter
• Identity resolution — merges anonymous and known identifiers into one profile — identity-resolution (no page yet), built on Union-find, Entity resolution and Hash join
• Delivery to destinations — retries, rate limits and parking of events that cannot be delivered — Exponential backoff with jitter, Token bucket, Dead-letter queue
• Destination authorisation — lets the platform call third-party APIs on a customer's behalf — OAuth 2 and OpenID Connect
• Identifier hashing — sends personal identifiers to ad platforms in hashed form — Cryptographic hashes
• Operations and assurance — keeps ingestion and delivery running through deploys and failures, protects acknowledged events and profiles, checks the input parser — Process supervision, Graceful draining, Mutual TLS, Primary-backup replication, Point-in-time recovery, Coverage-guided fuzzing, Deterministic simulation testing, hdr-histogram (no page yet)

 web SDK ─┐                           ┌────────────── per-destination queues ───────────────┐
 mobile  ─┼─► collection ──► durable  │ ─► transform ─► dedupe ─► identity ─► rate limit ─► │─► warehouse
 server  ─┤   (auth, schema)   log    │                                     retry/backoff   │─► ads API
 SaaS    ─┘      │                    │                                     dead-letter  ───┤─► email tool
                  ▼ reject (400)      └─────────────────────────────────────────────────────┘
              invalid event                                  ▲ profile store: id graph + traits

How the parts fit together

Collection endpoint

Each source (a website, an app, a server) has a write key that identifies it. Events arrive over HTTP as JSON, one call per event or in batches. The usual event vocabulary is small: identify says who a user is and records traits such as name and email; track records an action with properties; page and screen record views; group links a user to an organisation. The RudderStack documentation gives identify as the call that associates a visitor's actions with an identity, to be sent after registration, after login and when traits change (source: RudderStack event spec, Identify). The endpoint rejects a request with invalid JSON, an unknown write key or an improper size before anything else happens; the RudderStack gateway lists exactly these three cases (source: RudderStack architecture docs).

Beyond syntax, a CDP often checks each event against a tracking plan: JSON Schema validation compiles each schema once and then checks every untrusted document cheaply. The check has three possible outcomes: reject at the door with an error the sender sees, accept and flag for review, or accept and drop violating fields. Rejecting at the edge is safest for data quality; flagging is safest for not losing events. Most platforms let the customer pick.

Durable event log

The endpoint must answer fast and must not lose what it acknowledged. The pattern is: write the event to durable storage, then respond, then process. RudderStack's gateway "temporarily stores all received event data" in PostgreSQL before it acknowledges receipt and deletes the event only after it was delivered (source: RudderStack architecture docs). Other designs use an append-only log: a segmented log splits the log into files named by their first offset, so writes always go to the newest file and old files are deleted whole when retention expires.

The log decouples the producer from the consumers. Each consumer, for example the delivery worker of one destination, keeps its own consumer offset, one integer per partition that says where it stopped. If a destination is slow or down, only its offset lags; the others proceed. After a crash the consumer restarts from its last committed offset. The delivery guarantee is at least once: the offset is committed after the work, so a crash between work and commit replays a few events. Replay is why the next part exists.

Deduplication

Duplicates have many sources: a mobile SDK retries after a timeout when the first request in fact succeeded, a consumer replays after a crash, an operator re-sends a day of data. Each event carries a client-generated unique ID (a message ID), which acts as an idempotency key: the platform stores it for a window and treats a second event with the same key as a no-op. To keep this cheap at event rates, a Bloom filter answers "definitely new" for the vast majority of events with no storage lookup; only the "maybe seen" ones go to the exact store. The window has to be longer than the longest realistic retry or replay, or duplicates leak through.

Identity resolution

A visitor browses anonymously and gets an anonymous ID from the SDK; later they log in and an identify call links that anonymous ID to a user ID; they may also use another device, another browser, or buy in a shop with only an email address. Identity resolution (identity-resolution, no page yet) turns these identifiers into one profile by treating every identify or login as an edge between two identifiers and computing connected components. The data structure for this is Union-find: a disjoint-set forest with union by size and path compression that merges sets and answers "same person?" in near-constant time, incrementally, as events stream in.

Three design points decide whether this works in practice:

• Which edges to trust. An email or a login ID is a strong link; a shared IP address or device ID is weak and can merge two strangers (a shared family tablet, an office network). Platforms apply rules per identifier type and caps on how many identifiers one profile may collect.
• Merges are hard to undo. Union-find only unions. A wrongly merged profile needs the edge history to be split, so keep the edges, not only the merged result, and keep merge events auditable.
• Probabilistic matching. When no deterministic identifier exists, matching moves toward entity resolution: blocking to find candidate pairs, scoring pairs, and clustering. This is more flexible and less certain; it belongs in a separate, reviewable layer. A hash join on the identifier columns is the workhorse when profile tables must be joined to event tables in batch.

Delivery to destinations

Delivery is where most of a CDP's code lives, because every destination has its own API, rate limit, authentication and failure behaviour. The patterns are the same across them:

• Per-destination queues. Each destination gets its own queue and offset, so a failing destination only delays itself. RudderStack's router sends transformed events to destinations and keeps them in the database until delivery succeeds, so that it can retry on failures (source: RudderStack architecture docs).
• Rate limiting. A token bucket per destination and account keeps the call rate under the API's limit while letting short bursts through.
• Retry with backoff. On a timeout or 5xx the delivery waits and retries after a delay that grows geometrically with random jitter (exponential backoff with jitter), so that a recovering destination is not hit by every queued retry at the same moment.
• Failure parking. Events that fail permanently (4xx for a malformed payload, or too many attempts) go to a dead-letter queue with the error, so one poison event does not block the queue behind it, and an operator can inspect, fix and replay.
• Transformations. Each destination needs a payload in its own format. RudderStack runs this as a transformation module with code in JavaScript and also lets users add their own functions, for enrichment, filtering and removing personal data (source: RudderStack architecture docs).

Destination authorisation

Many destinations are accessed on behalf of a customer who owns the account there. OAuth 2 is the standard: the customer authorises the platform once through the authorization-code flow and the platform stores a refresh token and renews short-lived access tokens. Tokens are secrets and must be stored encrypted; a revoked or expired token turns every event for that destination into a failure that backoff cannot fix, so the platform should stop retrying, alert the customer, and keep the events until the token is fixed.

Identifier hashing

Advertising and some analytics destinations expect personal identifiers (email, phone) as hashes, normalised first (lower-case, trimmed) and hashed with SHA-256 (Cryptographic hashes). The hash is a matching key, not an anonymisation: anyone with a candidate email can compute the same hash, so hashed identifiers are still personal data and must follow the same consent and deletion rules.

Operations and assurance

The promise to a customer is that an acknowledged event is delivered at least once, so the operational blocks all protect the log and the workers around it. Primary-backup replication is how the durable event log survives a machine loss: one node orders the appends and ships them to a standby, and the acknowledgement rule decides what the endpoint may say to the SDK. Replying only after the standby has the event costs latency on every request but means a crash loses nothing that was acknowledged; replying after the local append alone is faster and can lose the last events on a crash. An external arbiter decides which node takes over. Point-in-time recovery protects the profile store: a bad identity-resolution rule can merge two customers into one profile, and the repair is to restore the store to the instant before the rule was released from a base backup plus the archived change log, then replay events from the log from that point.

Workers (collection endpoint, deduplication, identity resolution, delivery) run under Process supervision, which restarts a crashed worker with a delay and burst limit. On a deploy, Graceful draining makes the collection endpoint stop accepting new connections, append everything already received to the log and answer clients with a reconnect hint, and makes a delivery worker finish the destination calls in flight and commit its consumer offset before it exits; events not finished stay in the log and are redelivered, with the idempotency key preventing a double send. Links between the endpoint, the log, the workers and server-side sources use Mutual TLS so that a service identity, not a network address, decides who may append or read.

The collection endpoint parses JSON from SDKs and servers that the platform does not control, so its decoder and schema validator are the inputs for Coverage-guided fuzzing: mutated events are run through the instrumented parser, and any input that reaches a new edge is kept. Deterministic simulation testing runs endpoint, log, deduplication and delivery in one thread over a simulated disk, network and clock, with crashes between append and acknowledgement and destinations that fail half-way; the checked property is that every acknowledged event reaches each destination at least once and, with idempotency keys, takes effect once.

hdr-histogram (no page yet) records ingest-to-log acknowledgement time and log-to-destination delivery time per destination in log-linear buckets with bounded relative error, so a single slow destination or a slow percentile of customers is visible while the average stays flat.

Data and control flow

One event, end to end:

1. An SDK or server posts an event with a write key; the endpoint authenticates the source, checks syntax and size, and validates it against the tracking plan.
2. The event is written to the durable log, and only then does the client receive a 200.
3. A processor reads the log at its offset, deduplicates by message ID, enriches the event and updates the identity graph; a new identify unions two identifiers.
4. For each connected destination the event is transformed into that destination's format and put on its queue.
5. The delivery worker takes events within the token-bucket rate, calls the API, and on success commits its offset; on failure it retries with backoff, and after the limit moves the event to the dead-letter queue.
6. A warehouse destination usually receives events in batches (files or bulk loads) instead of one call per event.

Control plane: the configuration of sources, destinations and connections lives apart from the data path. RudderStack splits it into a control plane (web app plus configuration backend) and a data plane that receives, transforms and relays events (source: RudderStack architecture docs).

Scaling

1. Ingest is stateless apart from the log write, so it scales by adding endpoint instances; the log's partitions decide how many consumers can run in parallel.
2. Processing scales by partition. To keep per-user ordering, partition by user or anonymous ID; the identity graph then needs either a shared store or a partition on the resolved profile ID, which can change after a merge.
3. Delivery scales per destination and is limited by the destination's rate limits, not by the platform; the token bucket is the throttle, and slow destinations should not slow the pipeline for others.
4. Identity graph grows with the number of identifiers, not with the number of events; cap identifiers per profile and expire anonymous IDs that never link to anything.
5. Storage of the log is bounded by retention; the dead-letter queue must be monitored, because it grows silently.
6. Regional deployment: a RudderStack-hosted workspace processes and stores event data in its chosen region, for example US or EU (source: RudderStack architecture docs).

Failure modes

Real implementations

The alternative: writing events straight into the warehouse and running reverse ETL out of it keeps one source of truth and fewer moving parts, at the cost of minutes of latency and no real-time routing. A CDP earns its place when events must reach many destinations quickly, or when the customer wants one vendor to carry the delivery mechanics (retries, rate limits, per-destination formats).

Sources

• RudderStack documentation: architecture, open source overview, Identify call.
• RudderStack server: repository; Snowplow: repository; Jitsu: repository; Segment analytics-next: repository.