retail-brokerage

How it is built

Architecture overview

A retail broker lets individuals buy and sell securities from an app without being the exchange or the clearing house. It is the customer's agent: it takes an order, checks that the customer can afford it, sends it to a market maker or an exchange over a FIX session, records the fills, keeps the customer's cash and positions, and later settles with a clearing firm. Around that trading core sit the things a consumer product needs: live quotes on the phone, push notifications, funding transfers from a bank, tax lots, corporate actions, and a login that attackers cannot walk into. The one idea that makes it work is two clocks joined by a ledger. The real-time clock (orders, quotes, seconds) must be fast and tolerate partial failure; the settlement clock (confirmations, funds, statements, a one-business-day cycle in the US since 28 May 2024) must be exact and reconcilable. Every order is a persisted state machine, every state change that touches money posts to a double-entry ledger, and every request that can be retried carries an idempotency key, so a retry or a crash can never create or destroy money or shares. At scale the system is partitioned by customer: Robinhood describes sharding its brokerage backend so that each shard has its own application servers, database and deployment pipeline, with a routing layer that maps a request to a user and the user to a shard (Robinhood, 2021).

Parts:

• Edge and identity — authenticate the person, protect the account, rate-limit — Passkeys (WebAuthn), OAuth 2 and OpenID Connect, Token bucket, mfa (no page yet)
• Order service — owns each order's lifecycle — order-lifecycle-state-machine (no page yet), Idempotency keys, Optimistic concurrency control
• Buying power and risk — can the customer afford this, within house limits — Pre-trade risk checks, margin-liquidation (no page yet)
• Execution gateway — talks to venues and wholesalers — FIX protocol (session layer)
• Market data — normalise, distribute and display quotes — sequenced-multicast-feed (no page yet), Market data feeds, Pub/sub fan-out, WebSockets
• Ledger — cash, positions, fees, holds — Double-entry ledger, Money representation, largest-remainder-rounding (no page yet), Event sourcing
• Portfolio and tax — cost basis and corporate actions — tax-lot-accounting (no page yet), Corporate-action adjustment
• Funding and settlement — move money in and out, settle with the clearing firm — Payment state machine, Reconciliation
• Notifications — tell the customer — Mobile push delivery
• Operations — keeps accounts, orders and the ledger available, recoverable and measured — Primary-backup replication, Point-in-time recovery, Process supervision, Graceful draining, Mutual TLS, hdr-histogram (no page yet), Deterministic simulation testing, Coverage-guided fuzzing

 phone / web ──HTTPS (auth, rate limit)──► edge ──► router ──► customer shard N
     ▲  ▲                                                      ┌───────────────────────────────┐
     │  └── quotes (WebSocket) ◄── fan-out ◄── normaliser ◄── │ order service ─► buying power │
     │                              feed handlers ◄ exchanges │      │               │        │
     │                                                        │      ▼               ▼        │
     └── push (APNs / FCM) ◄── events ◄───────────────────────│   ledger ◄── fills ◄ gateway ─┼──► FIX ──► venue /
                                                              │ (cash, positions, lots)       │          wholesaler
                                                              └──────────────┬────────────────┘
                                          daily: confirms, positions ◄──────┘──────► clearing firm, bank (ACH)
                                                          reconciliation

How the parts fit together

Edge and identity

A brokerage account is a target. The edge terminates TLS, authenticates the person and applies limits before any trading code runs. Login uses a short-lived session or token (OAuth 2 and OpenID Connect) and, increasingly, Passkeys (WebAuthn) because they are phishing-resistant and the server stores only a public key; a second factor (mfa, no page yet) guards sensitive actions such as adding a bank account or withdrawing. A Token bucket per user and per IP limits request rates, with a stricter bucket on login and on order entry. Contract with the rest: every downstream call carries a verified user id and a request id; no service below the edge re-authenticates passwords.

Order service and the order state machine

An order moves through a small set of states: received, validated, accepted by the venue, partially filled, filled, cancel requested, cancelled, rejected (order-lifecycle-state-machine, no page yet). The service is the only writer of that state, and every transition is stored with a version, so two events racing for one order (a fill and a cancel request crossing on the wire) are serialised by Optimistic concurrency control: UPDATE … WHERE version = ?, retry on conflict. The client supplies an order id; the first arrival creates the order, any repeat returns it (Idempotency keys), because mobile networks drop responses and users tap twice.

The service never trusts the venue's timing. An order sent and not acknowledged is unknown, not failed: the state is resolved by asking the venue for status (FIX defines status and mass-status requests), not by resending, because a resend can execute twice. Same discipline as in the Payment state machine.

Buying power and pre-trade checks

Before an order is sent, the broker checks it against the customer's buying power and against house limits (Pre-trade risk checks). For cash accounts this means available cash minus holds. For margin accounts, US rules set the floor: Regulation T requires initial margin of 50 percent of the market value for most margin securities (12 CFR 220.12) and FINRA Rule 4210 sets a maintenance requirement of 25 percent of the current market value of margin securities; brokers often set stricter house requirements. The check reserves the worst-case amount (the order's quantity at its limit price, or at a collared market price) as a hold in the ledger, atomically with accepting the order. Without the reservation, two orders placed a millisecond apart both see the same cash and both pass. Where the broker gives customers market access, SEC Rule 15c3-5 requires risk-management controls in front of the order.

When equity falls under maintenance, margin-call and liquidation logic takes over (margin-liquidation, no page yet); it uses the same gate and the same ledger, and the design is the one in the risk-engine system page.

Execution gateway

The gateway holds FIX sessions to venues and market makers (FIX protocol (session layer)): each session has sequence numbers, heartbeats and a resend mechanism, and the gateway is the single place that knows how to translate between the broker's order model and each counterparty's dialect. The order service sends it a command with a stable client order id; the gateway sends NewOrderSingle, and execution reports come back and are turned into events (ack, partial fill, fill, reject, cancel confirm). Choosing where to send an order (best execution; FINRA Rule 5310) is a policy owned by the routing logic and is not a block yet.

Contract: at-least-once events, in order per order id. The gateway may deliver an execution report twice after a reconnect; the order service ignores a report whose execution id it has already applied. It is the same idea as sequence numbers on the wire, one layer up.

Market data

Quotes come from exchange feeds or a consolidated vendor feed, as sequenced messages (Market data feeds; the sequenced-multicast pattern is sequenced-multicast-feed, no page yet). A normaliser converts them to one internal quote model and keeps the latest quote per symbol. Getting it to millions of screens is Pub/sub fan-out: encode the update once, deliver it to every connection subscribed to that symbol, and give each connection a bounded queue with an explicit policy for slow clients. For quotes the right policy is conflation: replace the queued quote with the newest, because nobody needs the quote from two seconds ago. WebSockets carry the stream to the app; a fallback to polling helps when a network blocks them. Holding quotes in the order path would be wrong: orders do not read the UI's quote, they carry their own price and are checked against exchange-side controls.

Ledger

Cash, positions, holds, fees and pending settlements are accounts in a Double-entry ledger. A buy fill is one transaction: debit "pending settlement" and credit the customer's cash by the cost, plus the fee, and increase the customer's position; transaction postings sum to zero per currency, so money and shares only move. Amounts are integer minor units with an explicit exponent (Money representation). The ledger is append-only: a correction is a reversing entry.

Fractional shares expose a rounding problem. The broker aggregates fractional orders, buys whole shares and allocates them back to customers; the allocation must sum exactly to the total bought, which is the job of largest-remainder-rounding (no page yet): round each share down, then give the leftover units to the largest remainders, so no cent or sliver of a share appears or vanishes.

Portfolio, cost basis and corporate actions

The portfolio view is derived from the ledger. Tax reporting needs lots: each purchase is a lot with a date and a cost, and a sale closes lots in an order the customer or default rule chooses (tax-lot-accounting, no page yet). The IRS describes two methods for stock where lots are identifiable, specific share identification and first-in first-out (IRS Publication 550). Corporate actions (splits, mergers, dividends) change positions and costs on a given date: see Corporate-action adjustment for the factor table that adjusts history; applied to lots, a split multiplies the quantity and divides the per-share cost so the total cost is unchanged. They are announced by data vendors, applied in a batch on the effective date, and verified against the clearing firm's own positions.

Funding, settlement and reconciliation

Deposits and withdrawals are payments with a lifecycle, so they use the Payment state machine: initiated, pending, settled, returned. A bank transfer can be returned days after the broker credited the customer; the design must decide how much buying power to grant against pending funds and how to recover a loss. Settlement with the clearing firm happens on the settlement clock; since 28 May 2024 the standard US cycle is one business day after the trade date (SEC amendment to Rule 15c6-1, law-firm summaries in Sources). Every day, Reconciliation compares the broker's ledger with the clearing firm's positions and cash and with the bank's statements. A break is an open item with an owner.

Operations

Primary-backup replication. Each user shard's database (orders, outbox, ledger, lots) has a primary and a backup (Primary-backup replication). The local transaction that holds cash, creates the order and writes the outbox row is acknowledged to the app only once the backup has it, so after a failover the relay still sends every order that was accepted and never sends one that was not. The shard map is durable and cached by the router, so a promoted backup is found by the same lookup.

Point-in-time recovery. Customer cash and positions cannot be rebuilt from anywhere else, and mistakes happen at the edges: a corporate-action run applied to the wrong lots, a bad migration, a mass fee posting. Point-in-time recovery restores a shard to a chosen moment from base backups plus the log, and the restored ledger is compared with the clearing firm's files in end-of-day Reconciliation before the shard serves traffic again.

Process supervision. The order service, outbox relay, execution gateway sessions, feed handlers and end-of-day jobs are supervised processes (Process supervision). A restarted gateway logs on to each venue with its stored sequence numbers and requests a resend, and a restarted relay resumes from the outbox without duplicating an order because the client order id is stable.

Graceful draining. Market-data WebSocket servers and order-service instances drain at deploy: stop taking new connections and new orders, finish in-flight accepted orders through the outbox, ask app clients to reconnect and resubscribe, and close FIX sessions with a logout. A deploy during market hours then does not show customers a cancelled quote stream or an order accepted but never sent (Graceful draining).

Mutual TLS. The router, order service, buying-power service, gateway, ledger and the links to the clearing firm and funding providers authenticate each other with certificates (Mutual TLS). The gateway accepts order commands only from the order service, so a compromised edge node cannot place an order straight onto a venue session.

Latency percentiles. hdr-histogram (no page yet) holds the distribution of tap-to-acknowledged (edge, buying-power check, local transaction), of order sent to execution report per venue, and of quote age at the phone (feed handler to last write). The market open and a news spike hit the tail first, so p99 and max per stage show which hop is slow where an average would not.

Deterministic simulation testing. The expensive bugs are interleavings of orders, execution reports, reconnects and money movements: a fill arriving twice after a reconnect, a cancel racing a fill, a withdrawal racing a buy. Deterministic simulation testing runs order service, ledger, a fake venue and a fake clearing firm in one seeded process with injected duplicates, drops and crashes, checks that buying power never goes negative and postings sum to zero, and replays any failing seed.

Coverage-guided fuzzing. Coverage-guided fuzzing is aimed at what takes bytes from outside: execution-report and market-data decoders in the gateway and feed handlers, the clearing firm's file formats (trade confirmations, positions, corporate actions) and the order and amount parsing at the API edge.

Data and control flow

Buying 10 shares at market

1. App sends the order with a client-generated order id; the edge authenticates and applies the token bucket.
2. The router maps the user to a shard (a durable, cacheable mapping, as Robinhood describes) and forwards.
3. Order service claims the id (idempotency), validates the instrument and market hours, and checks buying power.
4. In one local transaction: hold the cash in the ledger, create the order as accepted, write an outbox event.
5. A relay hands the event to the gateway, which sends the order over FIX.
6. Execution reports return; each is applied once: order state moves, ledger moves hold to cost, position grows, lot created.
7. Events fan out to the app (WebSocket) and, if the app is closed, to push. Time goes to the venue round trip and the outbound network; the broker's own part is a few database writes.

A quote to a phone

1. Feed handler decodes a quote and updates the symbol's latest value.
2. Fan-out finds the connections subscribed to the symbol and enqueues the encoded update on each.
3. Connections write at their own pace; slow ones conflate or disconnect. Time goes to fan-out (one update times the number of watchers) and to the last network hop.

End of day

1. Clearing firm sends trade confirmations, positions and cash.
2. The ledger is compared to those files; breaks go to operations.
3. Corporate actions with effective date today are applied; lots are adjusted.
4. Settled trades turn pending settlement into settled cash; statements and tax documents are generated.

Scaling

1. Partition by customer. Orders, ledger, lots and notifications are per user, so a shard owns everything for its users and a failure stays in one shard. Robinhood reports going from one shard at the start of 2020 to ten in 2021 and moving from a single database; its peak was 100k requests per second in December 2019 and 750k in June 2020.
2. A thin cross-shard layer. Features such as search, watchlists across accounts or a family view need an aggregation layer that fans a request out across shards and merges (with a cursor that holds one position per shard).
3. Event streams. Consume the shared log per shard; filter on user, or give busy flows their own topic (Robinhood describes both) (Partitioned commit log).
4. Fan-out for quotes scales with watchers: share the encoded payload, conflate, drop slow clients, add tiers of relay servers.
5. Spikes at the open and in news events. Shed non-essential load first (charts, news) and protect order entry and cancel; rate-limit by bucket; pre-warm capacity.
6. Ledger throughput. Group commits, a single writer per account, hot accounts split into sub-accounts (the broker's own fee account is a hot spot).

Failure modes

Real implementations

The alternative: a fully disclosed or introducing-broker model hands the ledger, margin and settlement to a clearing firm through its API, and the broker keeps only the customer experience and order routing. It is faster to build and cheaper to run, at the price of control over margin, funding and settlement timing. The design above is the one a self-clearing broker runs.

Sources

• E. Wong, N. Ziebart, How we scaled Robinhood's brokerage system for greater reliability, Robinhood newsroom, 25 June 2021 (sharding the single database by user, routing layer, aggregation layer, Kafka consumed per shard, 100k and 750k requests per second). post
• 12 CFR 220.12, Regulation T, initial margin requirements. eCFR
• FINRA Rule 4210, Margin Requirements (25 percent maintenance on margin securities). rule
• FINRA Rule 5310, Best Execution and Interpositioning. rule
• SEC Rule 15c3-5, Risk Management Controls for Brokers or Dealers with Market Access. rule
• IRS, Publication 550, Investment Income and Expenses (cost basis methods). pub
• Davis Polk, SEC adopts T+1 settlement effective May 2024, 17 Feb 2023. client update
• Alpaca documentation, About Broker API. docs