marketing-automation

How it is built

Architecture overview

Marketing automation sends the right message to the right person at the right time and measures what happened. It keeps a store of contacts with their attributes and behaviour, groups them into segments (rules over that data), and runs campaigns or journeys: graphs of steps such as "wait two days", "if the email was opened", "send a push", "add to a segment". Every step that reaches a person goes through a delivery channel (email over SMTP, mobile push, SMS or webhook) that has to authenticate itself, respect rate limits and process bounces and unsubscribes. Every reaction (delivery, open, click, unsubscribe, purchase) returns as an event that updates the contact and may move it along the journey. The idea that makes it work is that segments and journeys are both functions over one event stream plus a contact snapshot: a segment is a stored query, a journey is a durable state machine per contact, and both are re-evaluated when an event arrives or a timer fires.

Parts:

• Contact and segment store — contacts, attributes, membership sets, fast filters — Columnar storage, Roaring bitmaps, Vectorized execution
• Event pipeline — ingests behaviour events, ordered per contact — Partitioned commit log, Consumer offsets, Primary-backup replication
• Real-time aggregates — counts, uniques, windows for triggers — Windowed stream aggregation, HyperLogLog, Sliding-window counters
• Journey engine — per-contact state, waits, timers — Durable workflow execution, Event sourcing, Hierarchical timing wheel
• Email channel — builds, signs and sends mail, handles bounces and unsubscribes — email-delivery-smtp (no page yet), smtp-protocol (no page yet), DNS message format and client lookups, SPF, DKIM and DMARC sender authentication, bounce-processing (no page yet), TLS 1.3
• Push channel — provider-gateway delivery to phones — Mobile push delivery, HTTP/2, JSON Web Token (JWT)
• Send discipline — rate limits, retries, exactly-once-looking sends — Token bucket, Exponential backoff with jitter, Idempotency keys, Leases
• Experimentation — subject-line tests and send-variant selection — A/B testing, Multi-armed bandits
• Operations — recovery, supervision, testing — Point-in-time recovery, Process supervision, Deterministic simulation testing, hdr-histogram (no page yet)

 web/app SDK ─┐                                        ┌─► segment store ◄─ import / CRM sync
 email events ├─► ingest ─► event log ─► aggregates ───┤        │ membership changes
 push receipts│    (auth)   (per-contact order)        │        ▼
 webhooks ────┘                       │                └─► journey engine ◄── timers (wheel)
                                      │                        │ "send step"
                                      ▼                        ▼
                                 reports / A-B        send queue ─► rate limiter ─► channel workers
                                                      (idempotent)                  ├─► SMTP (DKIM-signed)
                                                                                    ├─► push gateway
                                                                                    └─► webhook / SMS
                       bounces, unsubscribes, opens, clicks ◄──────────────────────┘ (events re-enter)

How the parts fit together

Contact and segment store

A contact is a row of typed attributes plus a stream of events. A segment is a boolean filter over both, and membership has to be known fast for two jobs: "who is in this segment now" (a send) and "did this event just change anyone's membership" (a trigger). Columnar storage with Vectorized execution suits scans such as "country = X and last open within 30 days"; dictionary-encoding (no page yet) and zone-maps (no page yet) keep low-cardinality columns and date ranges cheap, and Delta encoding with Bit packing and PFOR shrink ids and timestamps. The membership of a stored segment is best held as a Roaring bitmap over dense contact ids, because "in A and not in B" then becomes bitmap algebra, and a suppression list is just one more bitmap to subtract.

Small installations do not need any of this. Mautic's documentation describes segments as lists of contacts defined by filters on contact fields, behaviour and company fields, combined with And/Or, and says dynamic membership only changes when a scheduled command rebuilds the segment (mautic:segments:update), while static segments are filled by hand, import, forms or campaign actions. That is the same contract with a batch rebuild instead of event-driven updates; the cost is that a contact can be in a stale segment between rebuilds. Event-driven maintenance needs the next part.

Event pipeline

Events are appended to a Partitioned commit log keyed by contact id, so all events of one person stay ordered and one consumer owns that person at a time. Segmented log gives cheap retention and replay, Consumer offsets let segments, journeys and reporting each read at their own pace, and Primary-backup replication keeps acknowledged events across a machine loss. The alternative named in the graph for the log is a Durable job queue; a queue is simpler when each event is consumed once, a log is better here because several consumers (segments, journeys, reports) need the same events and a late consumer must be able to rewind. Zero-copy I/O and Fast hash functions (for partition choice) matter only at high event rates.

Contract: an event has a contact id, a type, a timestamp from the source (not the receiver), and an idempotency key, so a retried webhook or a duplicated open pixel cannot count twice (Idempotency keys).

Real-time aggregates

Triggers such as "three site visits in a week" or "opened none of the last five emails" need counters. Use Windowed stream aggregation with watermarks (no page yet) to say when a window is complete despite late events (mobile apps upload events minutes or days late), and keep the running state in an LSM-tree, whose write-heavy pattern fits per-contact counter updates. Unique reach per campaign is a HyperLogLog, and per-contact send caps ("at most two marketing messages a day") are Sliding-window counters. Both trade exactness for constant memory; frequency caps usually need to be exact for the one contact and approximate is fine for reports.

Journey engine

A journey is a graph; each contact has a position in it. Durable workflow execution makes that position survive crashes by recording each decision as an event (Event sourcing) and replaying it to rebuild state; record-replay (no page yet) is the debugging side of the same idea. Its alternative in the graph is DAG-based batch execution, which is how a nightly "run the campaign for everyone" job works. The batch model is simpler and is what Mautic's documentation describes: the campaign update and trigger commands (mautic:campaigns:update, mautic:campaigns:trigger) run on a schedule and execute due events. The durable model reacts in seconds and scales per contact, but needs a log and a timer service.

"Wait three days" is a timer. With millions of contacts waiting, a Hierarchical timing wheel fires them in O(1) per timer, and a Slab allocator keeps the timer and state objects compact. Waits are defined in the contact's local calendar ("Tuesday 9 am local time"), which needs calendar-date-arithmetic (no page yet): daylight-saving gaps and per-contact time zones are the usual source of mis-timed sends.

The journey engine never sends directly. It writes a "send step" job with an idempotency key built from (journey, step, contact, iteration) into the send queue, which is the seam to the channels.

Email channel

The email-delivery-smtp block (no page yet) covers the whole path: build the message, sign it, look up the recipient's mail exchanger through DNS, open a TLS 1.3 connection, speak SMTP (smtp-protocol, no page yet; the base specification is RFC 5321), and classify the answer. Three things are specific to marketing mail:

• Sender authentication. SPF, DKIM and DMARC prove the mail comes from the claimed domain. DKIM (RFC 6376) signs selected headers and the body with a key published in DNS; DMARC adds alignment between the visible From domain and the SPF or DKIM domain and tells receivers what to do on failure. The original DMARC RFC (7489) is marked obsolete and replaced by RFC 9989, 9990 and 9991, so new work should cite those.
• One-click unsubscribe. RFC 8058 requires one List-Unsubscribe header with an HTTPS URI and a List-Unsubscribe-Post header whose content is List-Unsubscribe=One-Click, both covered by the DKIM signature. The receiving mail client then sends an HTTPS POST to that URI without user input, so the endpoint must identify recipient and list from the URI (with an unforgeable token), process the unsubscription, and not redirect. The unsubscribe endpoint is a write path into the segment store's suppression list, and it must be as available as the send path.
• Bounces. bounce-processing (no page yet) turns delivery status notifications (RFC 3464 defines the format) and synchronous SMTP failures into hard bounces (suppress at once) and soft bounces (retry, then suppress after a limit). Mautic's documentation names a command that fetches monitored mail for bounce management (mautic:email:fetch); listmonk's site lists bounces among its analytics.

Push channel

Mobile push delivery is the same send-queue contract with a different transport: the provider gateways take HTTP/2 requests authenticated with a JWT or certificate, and return per-token results that invalidate stale device tokens. The failure handling is the same as for email.

Send discipline

Every channel worker pulls from the send queue through a Token bucket per sending identity (IP, domain, provider), because mailbox providers throttle new or suddenly busy senders and a burst gets the whole domain deferred. Temporary failures retry with Exponential backoff with jitter. A worker takes a Lease on a job so that a crash hands it to another worker, and the Idempotency key on the job means the lease timing cannot cause a second copy: at-least-once delivery plus idempotent sends looks like exactly-once to the recipient. listmonk's site describes the same shape, "multi-threaded, high-throughput, multi-SMTP e-mail queues" with sliding window rate limiting.

Experimentation

A subject-line or timing test assigns each contact to a variant with Deterministic bucketing (hash of contact id and experiment id, so the assignment is stable and needs no stored table) and Experiment layering keeps overlapping tests independent. Analysis uses Hypothesis testing with CUPED to cut variance, and bootstrap-confidence-intervals and sequential-hypothesis-testing (both no page yet) for ratio metrics and early stopping. When the goal is to send the best variant rather than to learn, Multi-armed bandits (with Bayesian rate estimation and a PRNG) shift traffic toward the winner as clicks come in. Choose A/B when the decision must be defensible and the metric is slow; choose the bandit when each send is cheap and the metric arrives within hours.

Operations

The contact store and event log are the system of record, so Point-in-time recovery (a Write-ahead log plus base backups in Object storage, recovery via ARIES recovery) must be able to restore to just before a bad import or a mistaken segment send. Process supervision, Health checking and Graceful draining keep long-running senders from dropping jobs on deploy; Mutual TLS secures internal calls between ingest, journey engine and channel workers. Deterministic simulation testing and Coverage-guided fuzzing target the two hardest correctness problems: the timer and retry logic of the journey engine, and the parsers (HTML templates, inbound bounce mail). Latency is recorded with hdr-histogram (no page yet) histograms for event-to-journey-step delay and for send-queue wait, because a triggered message (a cart reminder, a one-time code) loses its value when the tail grows, and the percentiles tell the send-discipline layer when to shed bulk sends to protect triggered ones.

Data and control flow

One triggered email.

1. A purchase event arrives at ingest, is authenticated, deduplicated by its key and appended to the log partition of that contact. Cost: one log append; order of milliseconds.
2. The segment consumer applies it to the contact snapshot and flips segment bits; the aggregate consumer updates windowed counters.
3. The journey consumer sees the contact enter the segment that starts a journey, records "entered" as an event and schedules the first wait on the timing wheel.
4. The timer fires; the engine checks frequency caps and suppression, picks the variant by bucketing, renders the template and enqueues a send job with its idempotency key.
5. A channel worker leases the job, takes a token from the bucket for the sending domain, signs with DKIM, adds the RFC 8058 headers and hands the message to the receiving server over TLS.
6. The outcome returns as an event (accepted, deferred, bounced); later opens, clicks and unsubscribes arrive the same way and go back to step 1. Most of the elapsed time is the wait itself, then the receiving server.

One bulk broadcast. A segment is frozen into a bitmap snapshot, split into slices, each slice enqueued as a batch of jobs, and workers drain them under the token buckets. The send rate, not the code, sets the duration: a million messages at 100 per second take about 2.8 hours (arithmetic, not a measurement), which is why warming a new sending domain gradually is part of the plan.

One recovery. Stop sends, restore to the timestamp before the fault, replay the event log from the stored offsets, then re-run suppression before resuming, so that people who unsubscribed in the lost window are not mailed again.

Scaling

1. Sending reputation and provider rate limits bind first, long before CPU. Lever: more sending identities spread across domains and IPs, per-provider token buckets, gradual ramp-up.
2. Segment evaluation grows with contacts times rules. Lever: columnar scans, bitmaps for stored segments, incremental maintenance on events instead of full rebuilds, caching of expensive behavioural filters.
3. Timer and journey state grows with contacts in flight. Lever: partition by contact id so each partition owns its own wheel; compact state objects; move cold journeys to storage.
4. Event ingest peaks when a campaign lands and everyone opens at once. Lever: more partitions, batching, acknowledging before aggregation.
5. Experiment analysis is batch and the cheapest to defer.

Failure modes

Real implementations

Sources

• RFC 8058, Signaling One-Click Functionality for List Email Headers: https://www.rfc-editor.org/rfc/rfc8058
• RFC 6376, DomainKeys Identified Mail (DKIM) Signatures: https://www.rfc-editor.org/rfc/rfc6376
• RFC 7489, DMARC (obsolete); replacements RFC 9989 https://www.rfc-editor.org/rfc/rfc9989, RFC 9990 https://www.rfc-editor.org/rfc/rfc9990, RFC 9991 https://www.rfc-editor.org/rfc/rfc9991. Original: https://www.rfc-editor.org/rfc/rfc7489
• RFC 5321, Simple Mail Transfer Protocol: https://www.rfc-editor.org/rfc/rfc5321
• RFC 3464, Delivery Status Notifications: https://www.rfc-editor.org/rfc/rfc3464
• Mautic documentation: campaigns https://docs.mautic.org/en/5.x/campaigns/campaigns_overview.html, segments https://docs.mautic.org/en/5.x/segments/manage_segments.html, cron jobs https://docs.mautic.org/en/5.x/configuration/cron_jobs.html
• listmonk: https://listmonk.app