robo-advisor

How it is built

Architecture overview

A robo-advisor manages many small investment accounts with software instead of an adviser per client. It learns the client's goal, horizon and risk tolerance, maps them to a target allocation (for example 70 % equity funds, 30 % bond funds), buys the funds, and then keeps the account near that target as markets move and deposits and withdrawals arrive. It also tries to reduce taxes (selling losers to harvest losses, choosing which lots to sell) and reports performance. The idea that makes it work is that the investment decision is made once per model portfolio, not once per client: a small number of model portfolios are built by an optimiser, and the per-account work is cheap, deterministic arithmetic (compare holdings to the model, trade the difference within rules). That turns millions of accounts into a nightly batch of independent, restartable jobs, where the engineering difficulty is correctness of money, tax lots and orders rather than model sophistication.

Parts:

• Profile and suitability — client questionnaire to risk score and model assignment (business rules, not covered by a block here)
• Portfolio construction — builds model portfolios from expected returns and a covariance matrix under constraints — mean-variance-optimization (no page yet), covariance-shrinkage (no page yet), quadratic-programming (no page yet), Interior-point method, Sparse direct solver, LU and Cholesky factorisation, Sparse matrices (CSR)
• Goal projection — probability of reaching the goal under the chosen model — Monte Carlo integration, PRNG: PCG and xoshiro
• Account ledger and tax lots — holdings, cash, cost basis per lot — Double-entry ledger, Money representation, tax-lot-accounting (no page yet), fifo-lot-consumption (no page yet), Corporate-action adjustment, calendar-date-arithmetic (no page yet)
• Rebalancing engine — computes trades per account — threshold-rebalancing (no page yet), largest-remainder-rounding (no page yet)
• Order management — turns trades into orders, tracks fills, allocates back to accounts — order-lifecycle-state-machine (no page yet), FIX protocol (session layer), Idempotency keys, Hierarchical state machine
• Workflow and scheduling — runs the nightly and event-driven jobs reliably — Durable workflow execution, Durable job queue, Dead-letter queue, Hierarchical timing wheel, Leases, Fencing tokens, record-replay (no page yet)
• Performance and reporting — returns, risk, statements — portfolio-performance-metrics (no page yet), Event sourcing
• Storage integrity — durable, checksummed, concurrent-safe — Write-ahead log, Fsync and direct I/O, CRC32C checksums, Optimistic concurrency control, Message sequence numbers

 client app ──► profile & goal ──► model portfolio (built offline by the optimiser, a few dozen)
      │                                         │ target weights
      │ deposits / withdrawals                  ▼
      ▼                       ┌──────────  rebalancing engine (per account, parallel) ◄── prices, corporate actions
 account ledger ◄─ fills ─────┤                 │ proposed trades (tax-aware, bands, whole/fractional units)
 + tax lots                   │                 ▼
                              │         trade aggregation ──► order management ──FIX/API──► broker / venue
                              └── allocation of fills back to accounts ◄── fills ◄───────────┘
 nightly workflow (durable jobs) drives every box above; events feed reports and performance

Two clocks run through the system. A batch clock (usually after the market closes or before it opens) re-evaluates all accounts; an event clock (a deposit, a withdrawal, a dividend, a corporate action) triggers work for one account at once. Both feed the same rebalancing engine.

How the parts fit together

Portfolio construction

The classic approach is Markowitz mean-variance optimisation (mean-variance-optimization, no page yet): choose weights that maximise expected return for a given variance, or minimise variance for a given return, producing the efficient frontier (Markowitz 1952, Sources). With constraints (weights non-negative, bounds per asset class, a turnover limit) the problem is a convex quadratic program, and quadratic-programming (no page yet) covers the solver choices: active set, ADMM, and the interior-point method, whose Newton steps solve a symmetric linear system by a sparse direct solver on CSR matrices using a Cholesky or LU factorisation. The inputs are the weak point. Sample covariance matrices from few observations are noisy and ill-conditioned, and the optimiser amplifies the noise; covariance-shrinkage (no page yet; Ledoit and Wolf 2004, Sources) blends the sample matrix with a structured target to give a stable, invertible estimate. Expected returns are even harder to estimate, so many advisors use equilibrium returns or blend views with them (Black and Litterman 1992, Sources) or skip the return forecast and use risk-based weights. The optimiser runs offline and produces a small set of model portfolios (a risk ladder from conservative to aggressive, plus variants such as a tax-efficient or socially-screened version). The contract with the rebalancing engine: a model is a versioned list of (instrument, target weight, drift band).

Goal projection

"A 74 % chance to reach your goal" is a Monte Carlo projection of the portfolio's value paths with the planned contributions and withdrawals, using return and volatility assumptions for the model. The random numbers come from a seeded generator (PRNG) so the number shown to a client does not change on every page load. Assumptions are product choices; they need to be stated and versioned, because a client-visible probability is a regulated claim in many places.

Account ledger and tax lots

Every account is a double-entry ledger of cash and holdings: a buy debits cash and credits shares in one balanced posting; a dividend or fee is another posting. Amounts are integer minor units (Money representation); share quantities have a fixed number of decimals when fractional shares are supported. On top of the ledger, tax-lot-accounting (no page yet) keeps the cost basis per lot (quantity, acquisition date, price). Sales consume lots by a rule, usually fifo-lot-consumption (no page yet), or by specific identification (the engine picks the lots with the largest loss or the longest holding period). Splits and spin-offs change lots, not history: Corporate-action adjustment keeps raw lots and applies cumulative factors. Dates matter for tax: holding periods and the 30-day window around a loss sale (below) need calendar-date-arithmetic (no page yet): time zones, month ends, the exchange's trading days. Where several processes touch one account (a deposit while the rebalance runs), optimistic concurrency control with a version on the account stops a job from acting on a stale snapshot.

Rebalancing engine

For each account the engine compares current weights with the model and decides whether and how to trade. threshold-rebalancing (no page yet) trades only when a holding drifts outside its band, and then only back to the edge of the band (or to target) rather than the whole way, which cuts trades and tax events. New deposits are first spent on the underweight holdings, withdrawals are taken from the overweight ones: this fixes much of the drift with no sale at all. The engine rounds to what can actually be bought: whole shares or fractional units, a minimum trade size, and a cash buffer for fees. When a total amount has to be split into integer units that must sum exactly (for example allocating a block trade's fills to accounts), largest-remainder-rounding (no page yet) gives the parts that add up with the smallest rounding bias.

Tax-loss harvesting is the main tax feature: sell a holding at a loss to realise the loss, and immediately buy a similar but not identical fund so the allocation is kept. The constraint is the US wash-sale rule: a loss is disallowed if substantially identical stock or securities were bought within the 30 days before or after the sale (26 U.S.C. section 1091, Sources). So the engine keeps a per-client, per-security list of "do not repurchase until" dates, applies it across all of the client's accounts (including reinvested dividends), and picks replacement funds from a mapping of near-substitutes. Rules differ by country; the engine's tax module is a replaceable component per jurisdiction.

Order management

The rebalancing engine emits proposed trades; the order layer turns them into orders. It aggregates trades from many accounts into block orders per instrument and side (fewer orders, better execution), sends them to a broker or venue (FIX protocol, or the broker's own API), tracks them through the order-lifecycle-state-machine (no page yet) as a hierarchical state machine, and when fills come back allocates them to the accounts pro rata, rounding as above, so each account receives its share at the block's average price. Every order carries an idempotency key derived from (batch id, instrument, side, account set), so a retry after a timeout cannot double-send; sessions to the broker use sequence numbers to detect loss. The contract: an order is never regenerated from account state alone after it was sent. The system first asks the broker what happened to the original.

Workflow and scheduling

The nightly run is thousands to millions of small independent jobs (one per account or per batch of accounts), and each takes several steps: snapshot account, compute trades, aggregate, send, wait for fills, allocate, post to the ledger, notify. This is what durable workflow execution is for: the workflow's progress is recorded as events, so after a crash it resumes at the last completed step, never repeating a step with an outside effect. The jobs are held in a durable job queue: workers take a lease on a job, a lost worker's lease expires and the job is retried, and a fencing token lets the ledger reject a write from a worker that has lost its lease without knowing it. Jobs that keep failing go to the dead-letter queue for a person to look at, not to an endless retry loop. Waiting ("wait until the market opens", "wait three days for the deposit to clear") is a timer, held in a hierarchical timing wheel. For audit and debugging, record-replay (no page yet) records every non-deterministic input (clock, prices, broker responses), so a day's decisions can be replayed to see why an account traded.

Performance and reporting

Reporting reads the event stream (Event sourcing): the ledger is the truth and the statements are projections. portfolio-performance-metrics (no page yet) covers the formulas: time-weighted return (strips out the timing of deposits, for comparing a portfolio with a benchmark), money-weighted return (what the client actually earned), and risk figures such as volatility and drawdown. Performance must reconcile with the ledger exactly; the usual bug is a deposit counted on the wrong day.

Storage integrity

Account data is money, so the store writes through a write-ahead log and flushes in a way that survives power loss (Fsync and direct I/O); records carry CRC32C checksums so corruption is detected on read rather than acted on.

Data and control flow

Nightly rebalance of one account:

1. The workflow starts with the account's version and a price snapshot; corporate actions since the last run are applied to the lots first.
2. The engine computes current weights and compares them with the model's bands. Inside the bands, nothing happens.
3. Outside: cash flows first, then trades back to the band edge. The tax module proposes loss-harvesting sales within the wash-sale list and picks lots.
4. The proposed trades are written to the trade store with the batch id; the account moves to "trades proposed".
5. At the aggregation step, trades of all accounts in the batch are summed per instrument and side into block orders; each gets its idempotency key and goes out.
6. Fills arrive (minutes, or the next open); each fill updates the block's state; the workflow allocates fills to accounts by pro-rata with exact rounding.
7. Each account's ledger gets balanced postings for shares, cash and fees, and lot rows are created or consumed; the version check ensures nothing changed meanwhile, otherwise the step retries from the snapshot.
8. Events are emitted: statement lines, client notification, performance update. Most wall time is waiting for the market; the compute per account is small.

A model portfolio update:

1. Market data and constraints change on a schedule (monthly or quarterly, or after a rule change).
2. The optimiser runs offline on shrunk covariances, producing a new model version.
3. Compliance reviews and approves; the new version is published with an effective date.
4. The next rebalance uses it: accounts now outside the new bands trade, subject to the tax rules, rather than all at once.

Scaling

In the order you would pull them:

1. Per-account work is independent, so scale out workers on the job queue; the limits are the shared price snapshot (read-only) and the ledger database's write rate.
2. Skip work: most accounts are inside their bands most nights; a cheap pre-check (drift test against the stored weights) avoids the full computation. Event-driven triggers cover the rest.
3. Aggregate before trading: block orders make order count proportional to instruments, not accounts; the allocation step turns blocks back into account postings in bulk.
4. Optimise rarely and centrally: the quadratic program belongs to the model, not the account. If per-account tax optimisation becomes a QP per account, restrict it to accounts with harvest candidates and warm-start the solver from the last solution.
5. Database: partition accounts by id; keep a ledger append path short; batch flushes. Large batches against one table are the usual bottleneck, not CPU.
6. Monte Carlo projections: cache by model and goal parameters; they do not depend on the account's exact holdings.

Failure modes

Real implementations

No complete open-source robo-advisor is listed here; commercial ones (Betterment, Wealthfront and others) are proprietary, and the open pieces above cover construction and workflow only. Ledger, tax-lot and order layers are built in house, usually on a relational database.

Sources

• H. Markowitz, Portfolio Selection, Journal of Finance, 1952. DOI
• O. Ledoit and M. Wolf, A well-conditioned estimator for large-dimensional covariance matrices, Journal of Multivariate Analysis, 2004. DOI
• F. Black and R. Litterman, Global Portfolio Optimization, Financial Analysts Journal, 1992. DOI
• B. Stellato et al., OSQP: An Operator Splitting Solver for Quadratic Programs, Mathematical Programming Computation, 2020. DOI, arXiv
• S. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press, 2004. free text
• 26 U.S.C. section 1091, loss from wash sales of stock or securities. Cornell LII