← Back
Home › Systems › Consumer applications

real-estate-listings

Shows homes for sale or rent on a map and in a list with filters, and alerts people when new listings match.

Zillow, Rightmove

Open the real-estate-listings pagereal-estate-listings CDN and edgecloud: CDNs cache content in hundreds of PoPs near users, cutting latency and egress bills; edge compute runs small code there. Bandwidth price spreads 10x between providers.›Hdr histogramdata-structures: Log-linear bucketed histograms with bounded relative error for latency percentilesDeterministic simulationtestingdistributed: Run the whole system in one thread under a seeded scheduler with simulated network, disk and clock, so every failure replays from its seed.›Incremental sync cursorsdistributed: The client keeps one opaque token; each pull returns only the changes after it plus the next token, so sync cost scales with what changed, not with the dataset.›Primary-backup replicationdistributed: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›Loan amortizationfinance: Annuity, interest-only and balloon schedules with exact rounding of the final paymentAddress geocodinggeometry: Parse, normalize and fuzzy-match postal addresses to coordinates and backAddress parsinggeometry: Split free-text addresses into components (house, street, city, postcode) with rules or a sequence modelGeospatial cell index (geohash,S2, H3)geometry: Map a lat/lon to a 64-bit cell id on a hierarchical grid, so "near" becomes an integer range scan in any ordinary sorted index.›Kriging spatial interpolationgeometry: Variogram-based kriging and IDW to interpolate sampled field values onto gridsLabel placement collisiongeometry: Place map labels and icons without overlap using a collision grid and prioritiesMap tile pyramidgeometry: Z/x/y tile pyramids, quadkeys and overviews for multi-resolution maps and imageryR-treegeometry: A balanced tree of nested bounding boxes that indexes rectangles and points, answering window and nearest-neighbour queries by pruning whole subtrees.›Vector tile encodinggeometry: Mapbox Vector Tile protobuf: zigzag-delta geometry commands, layers and feature attributesImage resampling filtersgraphics: Resize an image by convolving with a kernel (box, triangle, cubic, Lanczos) whose width scales with the shrink factor, done as two separable 1-D passes with precomputed weights.›Process supervisionhardware-os: A parent process starts services, reaps their exits and restarts them under a delay and burst-limit policy, then stops them with SIGTERM before SIGKILL.›Gradient-boosted decision trees(GBDT)machine-learning: Add small regression trees one at a time, each fitted to the loss gradient; histogram splits over 256 bins make it the strongest tabular default.›Learning to rankmachine-learning: Train a scorer on query-grouped data so that the order it induces maximizes NDCG; LambdaMART (boosted trees with NDCG-weighted pair gradients) is the production default.›Quantile regressionmachine-learning: Predicting quantiles with the pinball loss for probabilistic forecasts and prediction intervalsImage codecsmedia: JPEG (DCT, Huffman), PNG, HEIF/AVIF decoding and encoding fundamentalsEmail delivery smtpnetworking: Outbound SMTP: queues, MX lookup, retries, bounces, feedback loops and IP warm-upMobile push deliverynetworking: Sending notifications through APNs and FCM: HTTP/2 requests, token hygiene, collapse keys, TTL and priority, and the retry rules that keep you un-throttled.›Boolean expression indexingsearch-sort: Indexing conjunctive targeting expressions so one event finds all matching ads/subscriptions fastCoverage-guided fuzzingsecurity: Mutate inputs, run an instrumented target, and keep any input that reaches a new code edge; feedback turns blind guessing into incremental search.›Mutual TLSsecurity: TLS where the client also proves a certificate key, so every connection carries a verified, rotatable machine identity.›Entity resolutionstorage: Decide which records describe the same real-world thing: block to cut the pair space, score pairs with a probabilistic model, cluster the matches transitively.›Inverted indexstorage: Map each term to a sorted, compressed list of the documents containing it; the core of full-text search and other fast set queries.›Point-in-time recoverystorage: Periodic base backups plus a continuously archived change log let you rebuild the state as of any instant, then branch from there.›Secondary indexesstorage: Maintaining secondary and multikey indexes consistently with the primary dataGraceful drainingsystems: On a termination signal, stop taking new work, finish in-flight work until a deadline, tell clients to reconnect elsewhere, then exit; clients resume from a token.›HTTP caching semanticsnetworking: The rules (RFC 9111) that decide whether a stored response is still fresh, and how a cheap conditional request revalidates it when it is not.›Cooperative schedulingconcurrency: Green threads/coroutines yielding at await points on a user-space schedulerFault injectiondistributed: Inject latency, errors, kills and partitions via proxies, netem or hooksRecord replaydistributed: Record every nondeterministic input (time, randomness, I/O, model and tool outputs, per-tick player input) and re-execute deterministically for debugging, replays and state rebuildMessage sequence numbersdistributed: A per-stream counter on every message lets the receiver detect loss, duplicates and reordering, ack cheaply and resume after reconnect with "send from N".›Cursor paginationnetworking: Keyset/opaque-cursor pagination for stable incremental reads from APIs and tablesLeasesdistributed: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.›Health checkingsystems: Decide per backend whether to route to it using probes and real-traffic errors, with consecutive-count hysteresis, capped ejection and a panic fallback.›Address interpolationgeometry: Estimate a house-number position along a street segment from known address rangesFuzzy string matchingsearch-sort: Edit distance, Jaro-Winkler and trigram indexes for approximate name and address matchingNamed entity recognitionmachine-learning: Find the character spans of entities in text and give each a type, by labelling every token and decoding a valid label sequence.›Text normalizationmachine-learning: Rewrite messy text into one canonical form for matchers and models, or into spoken words for speech output (and back), keeping a map to the original offsets.›+ 43 moreTransparent interceptionnetworking: Redirect traffic to a local proxy (iptables TPROXY/REDIRECT, eBPF) without app changesSyscall interceptionsecurity: Intercept syscalls via ptrace, seccomp user notify or a user-space kernelFencing tokensdistributed: A number that grows with every lock grant and is checked by the protected resource, so a paused ex-holder's late write is rejected.›Byte-pair encoding (BPE)tokenizationmachine-learning: Learn a subword vocabulary by repeatedly merging the most frequent adjacent pair; byte-level BPE encodes any input losslessly with 32k-200k tokens.›Transformer blockml-training: The decoder-only pre-norm block (attention plus MLP on a residual stream) behind every modern LLM, with exact shapes, parameter, FLOP and KV-cache counts.›Convolutional codes and theViterbi decodersignal-processing: Streaming error-correcting code: each output bit is a parity of the last K input bits; the Viterbi decoder finds the most likely path through a 2^(K-1)-state trellis.›+ 24 moreEbpfhardware-os: Verified in-kernel programs on hooks (kprobes, XDP, tc) with mapsPtrace process controlhardware-os: Ptrace: attach, read and write registers and memory, single-step, intercept signalsSeccomp bpfsecurity: Syscall filtering with seccomp BPF programs+ 11 more+ 1 more

How it is built

Architecture overview

A listing portal shows homes for sale or rent on a map and in a list, filtered by price, size, features and place, and tells people when something new matches. The data is small by internet standards (listings, not billions of documents) but awkward: it comes from many sources with different fields, each listing changes status and price over time, every record is tied to an address that must be turned into a point, and the same property often arrives twice. The pipeline is ingest, clean, locate, index, serve. Sources, mostly MLS systems and broker feeds, are replicated incrementally; records are normalised and matched to one property; addresses are parsed and geocoded to a point and a cell; a search index handles words, facets and map boxes; the map is drawn from tiles and vector layers; photos are resized once and served from a CDN. The one idea that holds it together: the address-to-point step is the spine. A wrong point puts a house in the wrong neighbourhood, breaks duplicate detection and misplaces every valuation, so geocoding, property matching and the map all share one canonical place per property. Zillow and Rightmove are the closed examples; the data side has an open standard (RESO) and the geo side has open building blocks (PostGIS, Pelias, libpostal, Martin).

Parts:

 MLS / broker feeds ──► sync (cursor per source) ──► raw store ──► normalise ──► parse + geocode ──► property match
                              ▲                      (versions)    (fields,        (point, cell id)     (one id per home)
                              │ next cursor                       status)                                    │
                                                                                                             ▼
 photos ──► decode ─► resize ─► object store ─► CDN                                  listing store (primary + standby)
                                                                                   │ change stream
                       ┌───────────────────────────────────────────────────────────┼──────────────────────┐
                       ▼                                                           ▼                      ▼
              search index (words, facets, cell ids)                     valuation / ranking       saved-search matcher
                       │                                                    features + models             │
  user map pan ─► bbox → cells ─► query ─► rank ─► JSON + vector-tile pins ─► app                  email / push alerts

How the parts fit together

Feed sync

Listings are owned by MLSs and brokers. The industry standard for sharing them is the RESO Web API (OData v4) with the RESO Data Dictionary naming the fields; it replaces the older RETS transport, which RESO calls deprecated (source: RESO). RESO describes replication the way any mirror works: pull everything once, then keep asking for the latest changes (source: RESO, Web API page). That is an Incremental sync cursor per source: the portal stores one token (in practice the largest ModificationTimestamp seen, which the Data Dictionary defines as the time the MLS system last modified the listing) and each pull returns only records after it. Two details decide whether it is correct. The cursor must break ties: many records can share a timestamp, so it is the pair (timestamp, key) and the next page starts strictly after it. And a record can commit late, so each pull starts a little before the stored cursor and discards duplicates by version. Contract: every record is written to a raw, versioned store before anything else touches it, so a bad parser can be re-run without asking the MLS again.

Address pipeline

Feeds carry structured fields in a good case and messy text in a bad one (a unit in the street line, "St" and "Street", a postcode in the city field). address-parsing (no page yet) splits an address into house number, street, unit, city, region and postcode with rules or a sequence model; libpostal is the best-known open parser. Then address-geocoding (no page yet) normalises and matches it against reference data: address points, parcels, street segments with interpolated numbers, postcode centroids, in that order of precision. The result carries a precision class: rooftop, interpolated, street, postcode. Everything downstream reads that class, because a postcode centroid is fine for a region search and wrong for a pin. Parsing and fuzzy matching use the same text tools as search, and the reference data sits in an Inverted index of address tokens keyed to the cell index. The output is a point, a Geospatial cell index (geohash, S2, H3) id and the precision class.

Property identity

One home is listed by an agent, relisted after a price drop, syndicated by a portal and sold later. Entity resolution decides which records are the same property: block on geocoded address or parcel id plus unit, score pairs on address similarity, size, bedrooms and photos, and cluster. Why this matters more here than in most systems: price history, days on market and valuations all hang off the property id, not the listing id. Contract: a listing keeps its source id; the property id is an assigned cluster, and merges and splits are logged so a wrong merge can be undone.

Location index

Two location queries dominate. A map box (the viewport) is a set of cell ranges in the same sorted store as the other fields, because the cell hierarchy turns a rectangle into a few integer intervals; coarser cells give the cluster counts shown when zoomed out. A polygon (school zone, neighbourhood, a user-drawn boundary) needs a different structure: an R-tree over the polygons' bounding boxes finds candidate regions for a point quickly, and an exact point-in-polygon test finishes. Which to use is simple: cells for "points in a window", an R-tree for "which regions contain this point". Region membership is stored on the listing as ids at ingest, so "in school zone X" is a facet and not a query-time computation.

Search index and facets

The search index is an Inverted index whose terms include words (description, features), facet values (property type, bedrooms bucket, status), the cell ids at several resolutions and the region ids. Numeric ranges (price, area, year) use range structures over sorted values. Keeping facet counts, sort keys and the primary record consistent is the job of secondary-indexes (no page yet): when a price changes, the old index entries must go and the new ones must come, in one transaction on the store, or the facet counts drift. The index is a projection of the listing store's change stream, rebuilt from it on demand. Status is the hard field: "sold" or "under contract" must leave the default results within minutes, and a stale status is the most visible bug a portal has.

Ranking

Default order blends recency, price against the area and quality of the listing. Learning to rank trains on saves, tour requests and clicks, with Gradient-boosted decision trees (GBDT) as the production default (XGBoost documents its LambdaMART ranking objective; see XGBoost docs). Features come from the listing (photos, days on market), from the geography (price per square metre against the cell median) and from the viewer. Sort-by-price and sort-by-newest stay as pure sorts: users who ask for them expect them unmodified.

Valuation and affordability

An automated valuation is a regression on sales and listing features, again with GBDT. quantile-regression (no page yet) trains with the pinball loss to predict a low and a high quantile (XGBoost offers it as reg:quantileerror, source: XGBoost parameters), so the product shows a range, which is the honest way to display an estimate that is wrong by a margin. kriging-spatial-interpolation (no page yet) fits a variogram to the residuals and interpolates a smooth neighbourhood price surface, a feature for the model and a map layer. loan-amortization (no page yet) computes monthly payments from price, rate and term with exact final-payment rounding, so a search by monthly budget is a sort and a filter over a derived field, and the numbers shown match the lender's.

Saved-search alerts

Users save criteria (price 300-500k, 3+ bedrooms, inside a polygon). When a listing arrives or changes, the portal needs the set of saved searches it satisfies: reverse search. boolean-expression-indexing (no page yet) indexes the conjunctions so one event finds all matching subscriptions without scanning every user's search. Matching alerts are batched per user and sent by email-delivery-smtp (no page yet) or by Mobile push delivery, with a frequency cap so a hot market does not send twenty mails a day. The match runs off the same change stream as the search index, so alert and search agree on what is new.

Map and photos

The base map is a pyramid of tiles (map-tile-pyramid, no page yet) in raster or vector form (vector-tile-encoding, no page yet: the Mapbox Vector Tile format of protobuf layers and delta-encoded geometry). Listing pins are served as one more vector layer generated for the viewport, so the client draws pins with the same pipeline as roads. Price labels on pins need label-placement-collision (no page yet): a collision grid, with priority rules (favourites, newest) deciding which label survives. Tiles and photos are static and cacheable, so CDN and edge serves both with long-lived cache headers and versioned URLs.

Photos arrive as large JPEGs from agents. image-codecs (no page yet) decodes them; Image resampling filters produce a fixed set of sizes with a filter whose width scales with the shrink factor, so thumbnails do not alias; the renditions are written once to object storage and served from the CDN. Doing this on first request instead costs latency at the worst moment, the first view.

Operations

The listing store runs with Primary-backup replication and Point-in-time recovery: a parser bug that overwrites prices should be undoable to the minute before it ran. Services are supervised (Process supervision) and drained on deploy (Graceful draining); internal calls use Mutual TLS. Feed parsers read other people's data, so they are fuzzed (Coverage-guided fuzzing); the sync and merge logic suits Deterministic simulation testing. Latency percentiles per endpoint go into hdr-histogram (no page yet).

Data and control flow

A price change from MLS to screen:

  1. The sync job asks the source for records modified after its cursor and receives the updated listing.
  2. The raw record is stored as a new version; the normaliser maps fields to the canonical schema.
  3. The address is unchanged, so geocoding is skipped (cached by the normalised address); property match is by the existing id.
  4. The listing store commits; the change stream emits an event.
  5. The search index updates terms and facets; the matcher finds saved searches the new price now satisfies and queues alerts; the valuation features for the cell are marked for refresh.
  6. Time goes to the source's response and the batch cadence; downstream steps take seconds.

One map search:

  1. The viewport and filters arrive; the bounding box becomes cell ranges at a resolution chosen from the zoom.
  2. The index returns listings in range with facets and counts; at low zoom it returns cluster counts per coarse cell instead of points.
  3. The ranker orders the visible set and picks the pins to show; label collision decides which prices are drawn.
  4. The response is JSON for the list and a vector layer for pins, both cacheable for a short time per tile and filter.

A new address:

  1. Parse, normalise, geocode; the precision class is stored with the point.
  2. The point gives the cell id and region ids (school zone, neighbourhood).
  3. Property matching looks for an existing property at the same address and unit and either links or creates one.

Scaling

  1. Source throughput. Many sources, each slow and rate-limited. Levers: independent cursors per source, parallel pulls, incremental only after the first load.
  2. Geocoding cost. Each new address costs a lookup. Levers: cache by normalised address, local reference data, fall back to a coarser precision class and fix later.
  3. Query load on busy maps. The map pan generates many small queries. Levers: coarse caching per tile and filter set, cluster counts precomputed per cell, pre-rendered vector tiles for common views.
  4. Alert fan-out. One listing can match thousands of saved searches in a hot market. Levers: batching per user per day, frequency caps, expression indexing so matching is not a scan.
  5. Photo bandwidth. The largest egress is images. Levers: renditions at fixed sizes, modern codecs, CDN caching; bandwidth prices vary widely between CDN providers (source: the CDN and edge page).
  6. Index rebuilds. A new analyser or schema means a rebuild. Lever: build beside, swap, from the change stream.

Failure modes

FailureSymptomCauseFix
Stale sold listingshomes shown after they soldsync lag, status not propagated, cached pagesshort TTL on results, status events prioritised, treat missing-from-feed as closed after a grace period
Wrong pinhouse in the wrong street or cityambiguous address, postcode centroid used as a pointprecision class, show area instead of pin for coarse results
Duplicate listingssame home appears twicemultiple sources, no property matchentity resolution on address, unit and photos; log merges
Over-merged propertiestwo flats shown as oneunit missing from the addressrequire unit match for multi-unit buildings
Cursor gapsupdates missing from one sourceties on timestamp, late commits(timestamp, key) cursor, overlap window
Facet driftcounts do not match resultsindex update not atomicupdate terms in one transaction, rebuild from the change stream
Overlapping labelsprices unreadable in dense areasno collision handlingcollision grid with priorities, cluster at low zoom
Overconfident valuationestimate far from the sale pricethin data in the cell, no intervalquantile range, show confidence, fall back to a coarser model
Alert floodusers mute or unsubscribeno frequency cap, broad criteriabatch, cap, merge near-identical listings
Heavy photo pagesslow listing pages on mobileoriginals servedfixed renditions, modern format negotiation, CDN

Real implementations

namewhat it isrepolicencenotes
RESO Data Dictionary and Web APIthe real estate data standard and OData transportreso.org, RESOStandardssee reposdefines field names and the sync API
PostGISspatial types and indexes for PostgreSQLpostgis/postgisGPL-2.0 (see COPYING)polygons, R-tree style indexes, spatial queries
libpostaladdress parsing and normalisationopenvenues/libpostalMITthe address-parsing step
Peliasmodular open-source geocoder using Elasticsearchpelias/peliasMITgeocoding part
NominatimOpenStreetMap geocoderosm-search/NominatimGPL-3.0 (see COPYING)geocoding part
H3, S2cell indexesuber/h3, google/s2geometryApache-2.0location index
OpenSearch / Lucenesearch server and libraryopensearch-project/OpenSearch, apache/luceneApache-2.0search and facets
XGBoostgradient boosting incl. ranking and quantile objectivesdmlc/xgboostApache-2.0ranking and valuation
Martinvector tile server for PostGIS and other sourcesmaplibre/martinMIT or Apache-2.0 (see repo)pin layer
Tippecanoebuilds vector tilesets from large GeoJSONfelt/tippecanoeBSD (see repo)offline tile build
imgproxyon-the-fly image resizing serverimgproxy/imgproxyApache-2.0photo renditions
libvipsfast image processing librarylibvips/libvipsLGPL-2.1photo pipeline
Zillow, Rightmoveproduction listing portals——closed

Sources

  • RESO, Web API and Data Dictionary: Web API, Data Dictionary, field ModificationTimestamp (DD 1.7).
  • A. Guttman, R-trees: a dynamic index structure for spatial searching, SIGMOD 1984. doi
  • Mapbox, Vector Tile Specification. spec
  • H3 documentation: h3geo.org; S2 Geometry: s2geometry.io.
  • R. Koenker, G. Bassett, Regression Quantiles, Econometrica 1978. jstor
  • T. Chen, C. Guestrin, XGBoost: A Scalable Tree Boosting System, KDD 2016. paper; XGBoost documentation, Learning to Rank, parameters.
  • S. Whang, H. Garcia-Molina, C. Brower et al., Indexing Boolean expressions, PVLDB 2009. doi

Similar systems

Share the most building blocks with this one.

photo-backup 34 %maps-routing 33 %map-tiles 30 %direct-mail 29 %appointment-booking 29 %