ipaas

How it is built

Architecture overview

An integration platform connects SaaS applications to each other without the customer writing a service. A customer defines a flow ("when a deal is won in the CRM, create a customer in billing and post to chat") and the platform runs it for them, many thousands of times a day, against APIs it does not control. The product is the library of connectors and the runtime that makes a flow of unreliable steps behave like a reliable one.

The one idea that makes it work is that every external effect is a recorded step in a durable run. A trigger (an inbound webhook or a poll of a remote API) becomes a persisted event; the event starts a run; each step's result is written down before the next step starts; a worker that dies is replaced and the run resumes from the log, not from the beginning. Everything else follows from that: retries are safe because steps carry idempotency keys, slow or failing vendors are isolated by queues and breakers, and customer-written code runs in a sandbox because the platform executes strangers' logic next to other tenants' secrets.

The platform is two planes. The control plane stores flow definitions, connections, credentials and schedules, and serves the editor. The data plane ingests events, executes runs and delivers results. They share a database but scale separately: the control plane is read-mostly and small, the data plane carries the load.

Parts:

• Ingest — receive vendor webhooks, verify them, poll APIs that have none, dedupe — Webhook delivery, Incremental sync cursors, Event deduplication window, cursor-pagination (no page yet)
• Connector auth and secrets — OAuth connections, token refresh, key storage — OAuth 2 and OpenID Connect, Secrets management, Mutual TLS
• Run engine — executes a flow as a durable, replayable run — Durable workflow execution, DAG workflow execution, Event sourcing
• Queue and workers — hand steps to workers with leases, retries and a graveyard — Durable job queue, Dead-letter queue, Exponential backoff with jitter
• Outbound call policy — protect vendors and the platform from each other — Token bucket, Circuit breaker, Idempotency keys
• Mapping and expressions — turn one app's record into another's — template-engines (no page yet), Pratt parsing, JSON Schema validation, schema-inference (no page yet)
• Sandbox — runs customer code steps — Process sandboxing, Cgroups and namespaces
• Scheduler — cron triggers, wake-ups, delays — cron-scheduling (no page yet), Hierarchical timing wheel
• State and recovery — run history, flow versions, backups — Write-ahead log, Point-in-time recovery, Primary-backup replication
• Operations and assurance — drain on deploy, supervise, test — Graceful draining, Deterministic simulation testing, Coverage-guided fuzzing, Process supervision, hdr-histogram (no page yet)

 SaaS vendors                                      SaaS vendors
  (webhooks) ───►┌────────┐                     ┌─►(REST calls)
                 │ Ingest │──events──┐          │
  (polled) ◄────►│ + poll │          ▼          │
                 └────────┘     ┌─────────┐  ┌──┴─────────┐
                                │  Event  │  │ Outbound   │
   Control plane                │  store  │  │ policy     │
  ┌───────────────┐  flow defs  │ (WAL)   │  │ rate limit │
  │ flows, creds, │────────────►└────┬────┘  │ breaker    │
  │ schedules     │                  │       └──▲─────────┘
  └──────┬────────┘            ┌─────▼──────┐   │ step calls
         │ secrets             │ Run engine │───┘
         ▼                     │ (durable)  │──► Sandbox (code steps)
  ┌───────────────┐            └─────┬──────┘
  │ Secrets/OAuth │                  │ steps as jobs
  └───────────────┘            ┌─────▼──────┐
                               │ Job queue  │──► workers ──► DLQ
                               └────────────┘

How the parts fit together

Ingest: events in

Vendors tell the platform about changes in two ways, and the platform must support both. Push vendors call a platform URL; the receiver does three things only: verify the signature (hmac-request-signing (no page yet)), append the raw body to the event store, and answer 2xx. Doing more before answering invites the vendor's retry, so mapping and routing happen later from the stored event. This is the receiving half of Webhook delivery, and the dedupe rule is the same as the sender's: the vendor will redeliver, so the receiver keeps recent event ids in Event deduplication window and treats a second arrival as a no-op.

Pull vendors have no push, so the platform polls. The contract is the one in Incremental sync cursors: store one opaque cursor per (connection, trigger), ask for changes since it, process, and only then advance it. Advancing after processing gives at-least-once delivery; the dedupe step turns that into effectively-once.

cursor-pagination (no page yet) is how the poller reads a vendor's list endpoint when a change set spans several pages: it follows the vendor's opaque next-page token until the page is empty and only then moves the stored sync cursor, so a crash mid-listing re-reads pages instead of skipping them. Offset paging is avoided because records inserted during the walk would shift the pages; the pages go into the event store as ordinary events, with the dedupe step absorbing the overlap.

Run engine: a flow is a durable run

Two models compete. Durable workflow execution runs ordinary code and replays it from a log of completed step results after a crash; it suits flows with loops, branching on data and long waits. DAG workflow execution holds the flow as a graph of steps with declared inputs and releases a step when its inputs are done; it suits visual builders where the editor already draws a graph, and it makes parallel branches and "skip downstream of a failure" cheap to express. Platforms with a canvas editor tend to store a graph and execute it with a small replay log; platforms that expose code store the code. Both persist the run as an append-only history (Event sourcing), which doubles as the run-history screen customers debug with.

Long delays ("wait three days, then check") are timers, and millions of them are cheap in a Hierarchical timing wheel rather than a sleeping thread each. Scheduled triggers use cron-scheduling (no page yet); the hard parts are time zones, daylight-saving gaps and runs missed during an outage, so skip, run-once or run-all is a per-flow setting.

The contract between engine and queue: the engine emits "run step X of run R, attempt N" as a job and receives either a result or a failure. It never calls a vendor itself.

Queue and workers: where retries live

A step becomes a row in a Durable job queue. A worker takes it under a Leases visibility timeout, executes, and acknowledges; a worker that dies lets the lease lapse and another picks the job up. Failure policy is split by cause. Transient errors (timeouts, 429, 5xx) retry on Exponential backoff with jitter, honouring the vendor's Retry-After when given. Permanent errors (400, 401 after a refresh attempt, 404) stop retrying at once. After a bounded number of attempts the job goes to a Dead-letter queue with its reason, and the run is marked failed and visible to the customer, who can fix the data or credential and replay. Lease expiry means a step can run twice, so the queue's contract is at-least-once and the call policy below must make that harmless.

Outbound call policy: being a good citizen of other people's APIs

The platform makes calls for thousands of customers to a few hundred vendors, and most vendors limit per account. Rate limiting is therefore per connection, not per platform: a Token bucket per (vendor, connection) allows the vendor's burst and average,. When a vendor is down, a Circuit breaker per vendor stops workers from burning attempts and holding connections; open means jobs for that vendor wait in the queue instead of failing, which keeps the dead-letter queue for real errors.

The write side needs Idempotency keys. Creating an invoice twice because a lease expired is the failure customers notice. Where the vendor accepts a key, the platform sends a key derived from (run id, step id), which is stable across attempts. Where it does not, the step performs a lookup-then-create keyed on a natural field, and the connector documents the remaining window honestly.

Connections: OAuth and secrets

Most connectors authenticate with delegated access, so the platform is an OAuth client for hundreds of providers (OAuth 2 and OpenID Connect). It stores a refresh token per connection, refreshes access tokens ahead of expiry, and handles the refresh token rotating or being revoked, which surfaces as a "reconnect" task for the customer instead of a failing run. Tokens and API keys are encrypted at rest and decrypted only inside the worker for the duration of a call (Secrets management); the editor and logs only ever see a reference.

Not every connection is an OAuth grant. Some vendors and customers' private systems accept only a client certificate, and the platform's own internal hops (engine to queue, workers to secrets store) should not share bearer tokens. Mutual TLS covers both: the certificate and key are stored as a connection secret like any refresh token and presented at call time by the outbound client, and inside the platform each service's certificate identifies it, so a compromised worker cannot impersonate the engine.

Mapping: the data between two schemas

The editor's field mapping ("{{trigger.contact.email}}") is a small expression language compiled once per flow version. Pratt parsing covers the expression grammar; a template-engines (no page yet) layer renders strings with loops and filters. Before a call, the payload is checked against the connector's schema (JSON Schema validation), which turns a vendor's cryptic 422 into a local error that names the field.

When a connector has no published schema, schema-inference (no page yet) derives one from sample responses captured during connection testing: field names, types, which fields are sometimes missing or null, and which are arrays. The editor uses it to offer the field picker, and the validation step before a call compares live payloads against it, reporting drift as a mapping warning rather than a failed run.

Sandbox: customer code

Code steps (JavaScript, scripts) run strangers' logic beside the secrets of other tenants. Process sandboxing gives the shape: a deliberately powerless child process, a small privileged parent that brokers the few operations it may do (HTTP out, read its own input), and a narrow pipe between them. The child's limits come from Cgroups and namespaces (CPU, memory, pids, private network and filesystem view) and a syscall allow-list (seccomp-bpf (no page yet)). The platform assumes the sandbox will eventually be escaped, so the sandbox host holds no standing credentials; a microVM or V8 isolate trades start-up time against isolation strength.

State and recovery

The event store and run history are the system of record, so they follow database rules: a Write-ahead log before data pages, Primary-backup replication with a defined acknowledgement rule so a primary loss does not lose acknowledged events, and Point-in-time recovery for the day an operator or a bug corrupts data. Large payloads live in Object storage; the run history keeps a reference with the content hash. Retention is a product decision: replays are limited to retained runs.

Operations and assurance

Deploys must not cut runs in half. On SIGTERM a worker follows Graceful draining: stop leasing, finish or checkpoint in-flight steps, release the rest.

Deterministic simulation testing runs the queue, engine and a faked network and clock in one seeded thread, so "worker dies between the vendor call and the ack" is a reproducible test instead of a monthly incident, with fault-injection (no page yet) supplying the failures. Coverage-guided fuzzing targets the parsers that face the outside world: expression language, template engine and webhook payload decoders.

Process supervision restarts the receivers, engine and workers when they die; combined with leases on queued steps, a crashed worker's steps return to the queue without anyone being paged. hdr-histogram (no page yet) records the delay from webhook receipt to step start and the duration of each vendor call, so the 99th and 99.9th percentiles can be reported per connector and per tenant; that is how a slow vendor is told apart from a saturated queue.

Data and control flow

Flow 1: an inbound webhook becomes a vendor write

1. The CRM POSTs "deal won" to /hooks/<flow-token>. The receiver verifies the signature, appends the raw event to the store, and returns 2xx. Time here is one durable write, so it is bounded by the log's fsync (order of a millisecond on SSD).
2. The dedupe check drops it if the vendor's event id was seen in the window.
3. The engine loads the flow version that was current when the event arrived, creates a run, and enqueues step 1.
4. A worker leases the job, decrypts the connection's token, evaluates the mapping expressions, validates the payload against the schema.
5. The outbound policy takes a token from the (vendor, connection) bucket; if the breaker is open the job goes back to the queue with a delay.
6. The worker calls the vendor with an idempotency key (run id, step id), writes the response into the run history, and acks the job.
7. The engine reads the new history, releases the next step; the run ends when the last step completes. Most of the elapsed time is vendor latency, not platform work.

Flow 2: polling a vendor without push

1. The scheduler fires the trigger for a connection (every N minutes by plan).
2. A worker reads the stored cursor and requests changes since it, with the connection's rate limit applied.
3. For each returned record it writes an event keyed by the vendor id and version; duplicates collapse in the dedupe step.
4. Only after the events are durable does it store the new cursor. A crash before this step re-reads the same page, which dedupe absorbs.
5. If the vendor reports the cursor expired, the trigger falls back to a bounded rescan and tells the customer.

Flow 3: a failing step

1. The vendor returns 503. The worker records the attempt and releases the job with a backoff delay with jitter.
2. After repeated failures the circuit breaker for that vendor opens; other customers' jobs for that vendor wait instead of failing.
3. When the breaker closes, queued jobs drain at the bucket rate, not all at once.
4. A job that exhausts its attempts moves to the dead-letter queue; the run shows "failed at step 3" with the vendor's last response, and a replay button re-enqueues from that step.

Scaling

In the order you hit them:

1. Vendor rate limits, not CPU. Per-connection buckets and fair queueing across tenants stop one customer's backfill from starving the rest. The lever is shaping, not hardware.
2. Queue and run-history writes. Every step is several durable writes. Batch the log writes (group commit), keep payloads out of the hot rows by storing large bodies in object storage, and partition queues by tenant or flow so lease contention stays local.
3. Ingest bursts. A vendor that replays a day of webhooks arrives as a spike. The receiver does one append, so scaling is stateless horizontal nodes in front of the log, with the queue absorbing the burst.
4. Sandbox density. Process sandboxes start in milliseconds and share a kernel; isolates are denser still; microVMs cost more per tenant but isolate harder. Pick by trust in the code, not by throughput.
5. History growth. Retention windows and moving old runs to cold object storage keep the hot database small.

Failure modes

Real implementations

Commercial platforms of this kind, such as Zapier, Workato and MuleSoft, are not open-source; they are named only as examples of the category.

Sources

• Hohpe, Woolf, Enterprise Integration Patterns, Addison-Wesley 2003. site
• Standard Webhooks specification (signing and delivery conventions). repo
• RFC 2104, HMAC: Keyed-Hashing for Message Authentication. rfc
• RFC 6749, The OAuth 2.0 Authorization Framework. rfc
• Project documentation and LICENSE files of the repositories in the table above.