two-sided-marketplace

How it is built

Architecture overview

A two-sided marketplace connects people who offer something (hosts, sellers) with people who want it (guests, buyers) and earns a fee for making the match, the payment and the trust work. It is a small set of services around one shared catalogue. A listing service holds the supply and its availability. Search and ranking turn a query into an ordered short list. A booking or checkout service reserves one unit of supply for one buyer without double-selling it. A payments service takes the buyer's money, holds it, splits off the platform fee and pays the seller later. Trust services (identity, reviews, fraud, disputes) make strangers willing to transact, and experimentation decides every change. The idea that holds it together: the platform sits in both the ranking and the money flow, so it can hold funds until the service is delivered and can learn from every outcome (which listing was chosen, booked, delivered well) what to show next. Airbnb, Etsy and eBay are the examples; their internals are mostly proprietary, so this page uses published papers and public payment-platform docs.

Parts:

• Listings and availability — structured supply, media, price rules, calendar or stock — Object storage, inventory-reservation (no page yet), Optimistic concurrency control
• Search and ranking — retrieve candidates, rank them, diversify — Inverted index, BM25 ranking, Posting list intersection, Geospatial cell index, Learning to rank, Gradient-boosted trees, Two-tower retrieval, HNSW, Top-k selection, embedding-inference (no page yet)
• Booking and holds — reserve a unit for a buyer for a short time — inventory-reservation (no page yet), conditional-writes (no page yet), Hierarchical timing wheel, Idempotency keys
• Payments and ledger — authorize, capture, hold, split, pay out, refund — Payment state machine, Double-entry ledger, Money representation, Event sourcing, largest-remainder-rounding (no page yet)
• Trust and reviews — ratings, abuse limits, fraud signals — Bayesian average rating, wilson-score-interval (no page yet), Sliding-window counters
• Messaging — buyer-seller conversation and notifications — WebSockets
• Experimentation — randomised tests of every ranking and policy change — A/B testing, Experiment layering, Deterministic bucketing, CUPED variance reduction
• Operations and verification — keeps the booking path alive, authenticated and recoverable, and finds faults before users do — hdr-histogram (no page yet), Process supervision, Graceful draining, Mutual TLS, Primary-backup replication, Point-in-time recovery, Coverage-guided fuzzing, Deterministic simulation testing

 seller ─► listing service ─► listing store ──► index builder ──► search index ──┐
            (calendar, price)  (+ media in object store)          (inverted + vectors)
                                      ▲                                          ▼
 buyer ─► query ─────────────────────────────────────────► retrieve ─► rank ─► diversify ─► results
   │                                                                       ▲ features, models
   │ choose                                                                │
   ▼                                                                       │ logs: impressions, clicks, bookings
 booking service ─ hold (TTL) ─► payments ─ authorize/capture ─► platform balance ──► ledger
   │                                │                                 │ after delivery
   │ confirm / decline              └─ refunds, disputes              ▼
   ▼                                                            payout to seller
 messages ◄──► seller        reviews ─► trust signals ─► ranking features, fraud rules

The transaction loop (search, hold, pay, deliver, pay out) must be correct: no double sale, no lost money. The learning loop (log outcomes, train rankers, run experiments) must be fast. They share the listing and the logs, and little else.

How the parts fit together

Listings and availability

A listing is a structured record plus photos. Photos and documents go to object storage; the record lives in a transactional store, which is the source of truth for price and availability. Search never owns that truth: the index is a derived copy, rebuilt from change events, and a search result is a hint that a booking must re-check. For anything with a calendar or limited stock, the unit of contention is the listing-night or the stock item. A reservation must be an atomic check-and-set on exactly that unit (inventory-reservation, no page yet, built on conditional-writes, no page yet, or on optimistic concurrency: read the version, write if unchanged). Contention is per listing, so a global lock would serialise the whole site for no benefit.

Search and ranking

Search is two stages, because ranking every listing with a heavy model is too expensive.

• Retrieval narrows thousands or millions of listings to hundreds. Text and attribute filters use an inverted index with posting-list intersection and, for text, BM25; "near here" becomes a range scan over a cell id. Availability and hard constraints (dates, guests, price band) are filters, not scores. Where wording differs between query and listing, embedding retrieval helps: Etsy's paper on unified embedding retrieval says it helps tail queries, where query and listing words often do not match, and personalises broad ones from past interactions. Such a two-tower model puts queries and listings in one space and finds neighbours in an approximate index.
• Ranking orders the hundreds. The Airbnb KDD 2019 paper describes the path: a hand-written score, then gradient-boosted trees, whose gains later plateaued, then neural networks. The training label is the booked listing against the not-booked listings shown in the same search; the offline metric is NDCG with the booked listing as the only relevant item (see learning to rank). The first network (one hidden layer) only matched the trees; later models used a pairwise loss weighted by the NDCG change and about ten times more data (1.7 billion pairs). Listing-id embeddings overfit, because each listing has few bookings.

Choices that matter beyond the model:

• Position bias. Users click what is on top. Airbnb's 2020 follow-up adds position as an input during training, zeroes it randomly with probability 0.15 so the model does not lean on it, and sets it to 0 at serving; the paper reports +0.7 % bookings online.
• Cold start. A new listing has no bookings, so the engagement features the ranker likes are missing. The same paper treats this as estimating those features from nearby listings with the same capacity, rather than as an explore-exploit problem, and reports +14 % bookings for new listings. A rank boost for new listings had been tuned to be booking-neutral and was hard to tune. (A bandit approach, multi-armed bandits, is the alternative.)
• Diversity. A ranker scoring listings one by one fills the top with near-duplicates in price and location. Airbnb's diversity paper moves from heuristic rerankers to an LSTM that embeds the context of the whole result list, and reports it launched with +1.2 % online NDCG and +0.44 % bookings.
• Serving cost. In the two-tower variant the query tower runs once per search, and the paper reports a 33 % lower 99th-percentile scoring latency.
• Two sides. The ranker in these papers is trained on guest behaviour; host acceptance and cancellation are not discussed there. Hosts who decline need the supply side in the objective too.

The embeddings are used at serving time only: a pretrained model (embedding-inference (no page yet)) turns each query into a vector in the request path, and listing vectors are computed when a listing changes and stored in the approximate index. No training happens while a search runs; the model is replaced offline and the index rebuilt from the same change events as the text index.

Booking and holds

Selecting a listing starts a hold: the unit is reserved with an expiry (a lease on the calendar, expired by a timing wheel or a sweeper), payment is attempted, and the hold becomes a booking or lapses. For request-to-book the seller has a window to accept, and the payment stays authorised but uncaptured. Every request carries an idempotency key, so a retried tap on "Book" never books or charges twice. The hold must outlive checkout but be short enough that abandoned carts do not lock inventory.

Payments and ledger

A marketplace is not a merchant that takes money and keeps it: money moves buyer to platform to seller, minus fees, and sometimes back. The payments platform Stripe documents three Connect charge types: a direct charge on the seller's account; a destination charge made on the platform with funds moved to the seller at once and the platform fee kept; and separate charges and transfers, where the platform takes the payment first and moves money later, possibly to several accounts. The docs recommend the third for cases like a payment created before the seller is known or split across several parties, and describe holding funds on the platform balance until an event, such as a confirmed service, as the typical escrow-like pattern, while advising platforms to hold funds only with a clear purpose and to take legal advice. Airbnb's help page says transactions may be reviewed before they are sent, which can delay payouts up to 45 days after check-in.

The platform's own service on top:

• A payment state machine per order (authorized, captured, refunded, disputed), where a timeout means unknown, not failed, and retries reuse the same key.
• A double-entry ledger of every movement (buyer funds, platform fee, seller payable, refund), amounts as integer minor units (money representation). Splits, such as a fee or tax shared across line items, must sum exactly (largest-remainder-rounding, no page yet). Because the ledger is append-only, event sourcing fits: balances are folds over events.
• Reconciliation against the provider's reports, the truth for what actually moved.

Refunds and chargebacks are where the platform carries risk. In the Stripe docs, for destination charges and separate charges the refund or dispute amount is debited from the platform balance, and the platform may try to reverse the transfer to the seller; a seller balance can go negative. Hold policy, seller verification and reserves exist to bound that exposure.

Trust and reviews

Strangers transact because the platform vouches for them: verified identity, ratings, and arbitration in a dispute. Reviews are the central data and the most manipulated. Bolton, Greiner and Ockenfels study how reciprocity in feedback giving distorts reputation information on eBay-like platforms and how changing the information flow can make reputation more accurate; Fradkin et al. ran field experiments on Airbnb on bias and reciprocity in reviews (per the title). The engineering consequences:

• Show an average that does not jump on one rating: a Bayesian average pulls items with few ratings toward the global mean, and a lower-bound score (wilson-score-interval, no page yet) ranks by confidence.

• Rate-limit and score abusive accounts with sliding-window counters, then escalate to manual review.

Messaging

Buyer and seller talk before and after booking. A WebSocket channel pushes new messages; the message store is ordinary durable storage, and notifications fall back to email and push when no socket is open. Platforms want conversation on-site, for safety and as dispute evidence.

Experimentation

Every ranking, pricing and policy change ships behind an A/B test. Users are assigned by deterministic hashing of a stable id with a per-experiment salt, so the same person always sees the same arm; layers let independent experiments run on the same traffic. In a marketplace the arms share the same supply: a boosted listing in one arm takes bookings from another, so effects can leak between arms, and results need guardrail metrics for both sides. CUPED cuts variance using pre-experiment data so small effects (the papers above report sub-percent online gains) are detectable.

Operations and verification

The money and the calendar are the parts that must not be wrong, so the services around them are operated with care. hdr-histogram (no page yet) records latency percentiles of the search request and of the reserve-and-pay call, because buyers abandon on the slow tail and a hold with a short time-to-live is hurt by it more than by the average. Process supervision restarts the search, booking and payment services on a crash with a delay and burst limit, so a poison request does not loop. Graceful draining lets an instance finish in-flight reservations and payment calls before it leaves; the client retries with the same idempotency key and finds the first attempt. Mutual TLS authenticates the internal calls between booking, the ledger and the payment connector, so only those services can post money movements. Primary-backup replication keeps a standby for the transactional store that holds listings, holds and the ledger, with the acknowledgement rule chosen so a confirmed booking survives a primary failure. Point-in-time recovery restores that store to the instant before a bad migration or a faulty bulk price update, and the ledger is then re-checked against the payment processor. Coverage-guided fuzzing mutates listing uploads, search queries and payment-provider webhooks that the services parse, keeping inputs that reach new code. Deterministic simulation testing replays booking races, hold expiry and payment retries under a seeded clock and failure schedule, so a double booking found once replays from its seed.

Data and control flow

One booking:

1. The buyer searches. The query is parsed, filtered by dates, guests and place, and the candidate set is retrieved from the index (text, geo cells, embedding neighbours).
2. A model scores the candidates with query, user and listing features; a diversity step reorders the top; the results and what the buyer does are logged with positions.
3. The buyer opens a listing and selects dates. The booking service checks the calendar in the source of truth and writes a hold with an expiry using a conditional write; the index may have been stale, so this step can fail.
4. Payment is authorised (or captured) with an idempotency key; the state machine and ledger record each step.
5. The seller confirms automatically or manually. On confirmation the booking is final and messages open; on decline or timeout the hold is released and the authorisation voided.
6. After delivery (check-in, shipment, completion) the platform releases funds to the seller, minus fees, under its hold and review policy; the ledger posts the transfer and payout.
7. Both sides are asked for reviews; outcomes flow back into training data.

One ranker release:

1. Logs of impressions, clicks and bookings become training pairs with position as a feature.
2. A candidate model is trained and compared offline by NDCG on held-out searches.
3. It ships to a slice of traffic under an A/B test with booking, revenue and supply-side guardrails.
4. If the test wins it becomes the default; its logs are the next training set.

Scaling

Limits in the order they usually bite:

1. Search latency and freshness. Query volume and listing churn both hit the index. Levers: shard the index, rank in two stages so the heavy model sees hundreds, not millions, score the query tower once per search, cache popular queries, and update availability through a fast path separate from slow text fields.
2. Ranking quality per unit of data. Positions and cold start bias the logs. Levers: position-aware training, estimates for new listings, listwise diversity, then bigger models only when more data justifies them (Airbnb's final network was trained on about ten times more data).
3. Hot items in booking. Contention concentrates on a few popular listings or on-sale items. Levers: per-item conditional writes, short holds, queues for sales, and no cross-item locks.
4. Payments throughput and risk. Levers: idempotent retries with backoff, async payouts, reserves, and a ledger that reconciles against provider reports.
5. Experiment power. Limited traffic per market, plus interference: variance reduction, layering, market-level randomisation.
6. Trust operations scale with growth: rate limits, risk scores before payout, reviewer queues.

Failure modes

Real implementations

The alternative: a single-sided store (the e-commerce system itself) owns its stock, sets prices, and needs no payout, seller-trust or supply-side logic. An auction marketplace changes the matching step from ranked search to bidding; see the online-auction system. Pick two-sided when you do not own the supply.

Sources

• M. Haldar et al., Applying Deep Learning To Airbnb Search, KDD 2019 (arXiv 1810.09591): the model path from trees to neural networks, labels, NDCG, 1.7 billion pairs. paper
• M. Haldar et al., Improving Deep Learning For Airbnb Search, arXiv 2020: position bias, cold start, two-tower ranking. paper
• M. Abdool et al., Managing Diversity in Airbnb Search, arXiv 2020. paper
• R. Jha et al., Unified Embedding Based Personalized Retrieval in Etsy Search, arXiv 2023 (to appear at FMLDS 2024). paper
• M. Grbovic, H. Cheng, Real-time Personalization using Embeddings for Search Ranking at Airbnb, KDD 2018. Listed; not read. doi
• G. Bolton, B. Greiner, A. Ockenfels, Engineering Trust: Reciprocity in the Production of Reputation Information, Management Science 2013. doi
• A. Fradkin, E. Grewal, D. Holtz, M. Pearson, Bias and Reciprocity in Online Reviews: Evidence from Field Experiments on Airbnb, ACM EC 2015. Listed; abstract not read. doi
• J.-C. Rochet, J. Tirole, Platform Competition in Two-Sided Markets, Journal of the European Economic Association 2003, on how a platform's price structure across the two sides matters. Listed; not read. doi
• Stripe, Connect: how charges work and Account balances (charge types, holding funds, refunds, disputes, negative balances). charges, balances
• Airbnb Help Centre, payout help article 425 (payout review and processing times). page
• Repositories in the table above, README and LICENSE files read.