forum-qa

How it is built

Architecture overview

A forum or Q&A site stores user-written posts in a tree (a thread, its comments, replies to replies), lets readers vote on them, and orders everything by a score so the good content floats up. Reads outnumber writes by orders of magnitude, so the design is a small, strongly consistent source of truth (posts, comments, votes) with many cheap derived views around it: a sort order per listing, a search index, a page cache, vote counters. Votes are the hot spot: a few threads take most of the traffic and every visit may add a vote, a comment or a report. The idea that makes it work is to treat the ranking score as a pure function of a few numbers (votes, age, vote counts) that can be recomputed and indexed cheaply, and to keep everything expensive (search, similarity, spam models) off the posting path or behind a queue. Reddit's archived code, Lemmy and Discourse are the open references.

Parts:

• Content store — posts, comments, the comment tree, vote rows — closure-table (no page yet), MVCC, Primary-backup replication
• Input safety — turn user markup into safe HTML before it is stored or served — html-sanitization (no page yet)
• Voting and counters — record one vote per user per post and aggregate it without hot rows — Sharded counters, Sliding-window counters, Token bucket
• Ranking — hot, best, controversial, top-of-period sort keys — time-decay-scoring (no page yet), wilson-score-interval (no page yet)
• Search and related questions — find existing posts by words or meaning — Inverted index, BM25 ranking, embedding-inference (no page yet), HNSW
• Duplicates, spam and abuse — keep copies, spam and voting rings out of the ranking — Near-duplicate detection with MinHash LSH, Naive Bayes classifier, graph-community-detection (no page yet)
• Serving tier — page and fragment caches, edge caching for logged-out readers — W-TinyLFU, CDN and edge
• Operations — keeps threads and votes safe, fast and tested — Point-in-time recovery, Process supervision, Graceful draining, Mutual TLS, hdr-histogram (no page yet), Deterministic simulation testing, Coverage-guided fuzzing

 browser ─► CDN / edge ─► load balancer ─► web tier ─► L1 cache ─► L2 cache (Redis) ─► DB (source of truth)
 (logged-out pages)                           │  ▲                                       │ posts, comments,
                                              │  └─ sort keys, counters ◄── vote worker ◄┘ votes (one row per user+post)
   write: post / comment / vote               │
   ───────────────────────────► sanitize ─► rate limit ─► DB ─► queue ─► indexer ─► search index (BM25 + vectors)
                                                                    └─► spam / duplicate checks, ranking updates

How the parts fit together

Content store and the comment tree

Everything else can be rebuilt from the database, so it is the single source of truth. Stack Overflow's engineers describe exactly this: SQL Server is the source for all data, and everything in Elasticsearch and Redis is derived from it (Craver, 2016). A relational store with MVCC lets readers see a consistent snapshot while votes and comments keep arriving, and Primary-backup replication puts reads on replicas. Its replicas are asynchronous, one in a second data centre, so a replica read can be slightly stale.

The hard part is the tree. A comment thread has to answer: give me the whole subtree under comment X, sorted, with depth, and add a leaf without rewriting the tree. Three options:

• Adjacency list (each comment stores its parent): trivial to write, needs a recursive query or many round trips to read a subtree.
• Path or closure table (closure-table, no page yet; each row links an ancestor to a descendant with a depth): a subtree is one indexed range read, at the cost of one row per ancestor on every insert.
• A tree blob per thread: one read returns the whole structure, but every write is a read-modify-write of the blob.

Reddit's archived source documents this very trade-off. Its comment tree is stored as one value per thread in permacache (a persistent key-value cache), and adding a comment takes a lock on that thread for the read-modify-write. The source comments record two attempts to move the tree into Cassandra columns that were abandoned: one used counters as column values and "ended up being unreliable", the other produced "unexpected GC problems after longer time periods and generally insufficient regular-case performance" (reddit-archive/reddit, comment_tree.py). The lesson for a new design: pick the tree encoding by the write rate of your busiest thread, not by the average.

Contract to the neighbours: the content store emits a change event per write (new post, new comment, vote, edit, delete) that the indexers, caches and spam checks consume. Stack Overflow's search indexer polls the database for rows changed since a stored high-water mark using a SQL Server ROWVERSION column compared against a "last position" document in Elasticsearch (Craver, 2016); a commit log such as Partitioned commit log is the alternative when many consumers need the same stream.

Input safety

Users write Markdown or limited HTML, and one unescaped tag turns a thread into a script injection for every reader. html-sanitization (no page yet) belongs on the path where markup is rendered: parse the input into a tree, keep an allowlist of tags, attributes and URL schemes, and emit the result. The OWASP cross-site-scripting cheat sheet recommends output encoding when data is shown exactly as typed, names HTML sanitization for the cases where users may write markup, and warns that sanitizer libraries must be patched regularly because bypasses keep being found. Keep the user's source text and the sanitized output as two fields so the rules can be tightened later and old posts re-rendered.

Voting and counters

A vote is a fact: user, post, direction, time. The database enforces one row per user and post (a unique key), which also makes retries safe. The count is a derived number, and the naive UPDATE ... SET score = score + 1 turns a viral post into one row that every writer queues on. Two standard fixes:

• Sharded counters split the hot counter into N cells and sum them on read; increments no longer collide.
• Write the vote row only, and fold counts into the post row in batches by a worker. Readers see a score that is a few seconds old, which nobody notices.

Abuse limits sit on the same path. A Token bucket per user bounds votes, posts and reports per minute while allowing a burst; Sliding-window counters give the "how many in the last hour" numbers that gate privileges and detect floods. A new account that can vote at once is a spam and brigading tool, so many sites gate privileges on an earned trust score; Reputation scoring is the general block for turning good and bad events into such a score.

Ranking

Three sort keys cover almost every forum, and each is a small formula over counts and time. Because they depend on a few numbers, the site stores the result as an indexed sort key and listings become a range scan.

Hot (time decay, time-decay-scoring, no page yet). Reddit's archived _hot function is sign * log10(max(|s|, 1)) + seconds / 45000, where s is ups minus downs and seconds counts from a fixed date in 2005 to the post's creation time (reddit-archive/reddit, _sorts.pyx). Read it as: every factor of ten in net votes is worth 45,000 seconds, which is 12.5 hours (arithmetic), so a post with ten times the votes outranks a post that is half a day newer. The first ten votes count as much as the next ninety. Lemmy documents a variant with a gravity exponent: rank = scale * log(max(1, 3 + score)) / (time + 2)^gravity, default gravity 1.8, and a "scaled" variant that divides by the log of the community's monthly active users so small communities are not buried (join-lemmy.org, ranking algorithm). Lemmy's notes record the failure these variants fix: a ranking that ignores the age of the thread gives early comments an overwhelming advantage, and new comments pool at the bottom.

Best (confidence). Sorting by average rating or by upvotes minus downvotes is wrong for small samples: an item with 2 up and 0 down beats one with 100 up and 1 down on average, and a 55 % item with many votes beats a 60 % item with fewer on net score (Miller, 2009). The fix is the lower bound of the Wilson score interval (wilson-score-interval, no page yet): "given the ratings I have, there is a 95 % chance that the real fraction of positive ratings is at least what?" Reddit's confidence sort uses the same idea with a z-value for 80 % confidence (_confidence in _sorts.pyx). Use it for comment order and answer order, where age matters little. The same bound with a prior is the idea of Bayesian average rating.

Controversial. Large vote count with a near-even split; Reddit's function raises the total to the power of the balance.

Search and related questions

A Q&A site lives on search: most visits start from a search engine, and the site's own box must find the existing answer before someone asks a duplicate. Stack Overflow runs a plain full-text search on Elasticsearch for /search, to compute related questions and to suggest questions while a user is asking, with one index per site (Craver, 2016); an Inverted index scored with BM25 ranking over title, body and tags is the baseline. Embeddings of titles and bodies, produced by embedding-inference (no page yet) (a pretrained model run at serving time on each new post and each query, never trained here), in an HNSW index find paraphrases that share no words; Hybrid search and rank fusion merges the two lists. Lemmy ships a smaller version of the same: a "similar post search" while composing a new post, aimed at question and answer communities (Lemmy README).

Duplicates, spam and voting rings

• Duplicate posts. Near-duplicate detection with MinHash LSH catches the same text posted again with small edits; embeddings catch the paraphrase. On Q&A sites duplicates are usually closed by humans and linked, so the model's job is to propose candidates.
• Spam. A Naive Bayes classifier on words and link domains is the cheap first filter, trained on moderator decisions and updated by adding counts. Stack Exchange's community-run SmokeDetector bot is the documented counter-example: it reads new posts from the realtime feed and flags them with a long list of rules (regular expressions, keyword and domain blacklists, username checks) and reports them to chat rooms for humans (Charcoal-SE/SmokeDetector).
• Voting rings and sock puppets. Votes form a graph from voters to authors. Groups that vote almost only for each other show up as dense communities (graph-community-detection, no page yet). This is the standard graph approach to the problem, applied here by reasoning; none of the sources above describes it for these sites.

Serving tier

Most traffic is anonymous readers on a few hot pages, so caches carry the load. Stack Overflow's description is the clearest public one: a layer-1 cache inside each web server and a layer-2 cache in Redis; on a double miss the web server loads from the source, writes both layers, and a publish/subscribe message tells other servers to drop their layer-1 copy on a removal (Craver, 2016). W-TinyLFU is the modern admission policy for such a cache when popularity is heavy-tailed: a one-hit thread does not evict a hot one. CDN and edge serves the logged-out copy of popular pages; Cache invalidation covers keeping vote counts and edits from being served stale.

Operations

Point-in-time recovery complements the standby: replication copies a mistaken mass delete or a vote-purge bug to the standby within seconds, so the content store also keeps base backups and an archived change log. A wrongly purged thread or a moderator's bulk action is undone by restoring to the minute before.

The web tier, vote workers, the indexer and the spam checks run under Process supervision; a vote worker that dies is restarted from the queue, and since votes are one row per user and post, replaying a vote does not count it twice.

A deploy uses Graceful draining: web servers stop taking new requests behind the load balancer and finish the posts and votes in flight, and the vote worker drains its queue batch, so no accepted vote or comment is lost during a release.

The web tier, the cache, the database, the indexer and the spam service call each other over Mutual TLS, since they carry private data such as email addresses and moderation notes and only services with a certificate may write posts or read them.

Readers mostly see cached pages, so the latency that matters is the tail of cache misses, of the search box and of a vote becoming visible. These are recorded as an hdr-histogram (no page yet) with percentiles, and the tail shows hot threads that outgrow their cache entries.

Voting, ranking updates and the comment-tree writes are tested with Deterministic simulation testing: a simulated clock and queue replay duplicate and concurrent votes, a post deleted while it is being voted on and crashes between the database write and the counter update, and the check is that counters match vote rows. Failures replay from their seed.

The markup sanitiser, the search query parser and the comment-tree and vote endpoints read text from anyone, so they run under Coverage-guided fuzzing with mutated markup and requests, keeping every input that reaches new code.

Data and control flow

Posting a comment

1. The browser sends Markdown. The web tier authenticates, checks the rate limit (token bucket per user), and sanitizes the output form.
2. One database transaction inserts the comment, links it into the tree, and writes a change event. Lock scope is the thread, not the site; this is where a hot thread contends.
3. A worker consumes the event: spam score (Naive Bayes plus rules), duplicate check, notifications to the parent author, and an update to the thread's comment count.
4. The indexer adds the comment to the search index. Most of the delay between "posted" and "findable" is here, seconds not milliseconds.
5. The cache layers drop the thread's rendered fragment; a pub/sub message clears the other servers' in-process copies.

Casting a vote

1. Rate limit, then insert or update the single (user, post) vote row.
2. Add to the sharded counter; the post's ups and downs are folded in by a worker.
3. The worker recomputes the hot and best sort keys for the post and updates the listing indexes. Reads of a listing are then an indexed range scan, not a computation.
4. A live counter push (websocket) updates open pages. Stack Overflow pushes vote counts, notifications and new answers over websockets, with about 500,000 concurrent connections at peak in 2016 (Craver, 2016).

Reading a thread

1. Edge cache hit for logged-out users; otherwise the web tier checks the layer-1 then layer-2 cache.
2. On a miss it reads the post, the comment subtree (one range read with a closure table, or one blob with a tree blob) and the vote counts for the viewer, then renders HTML.
3. The viewer's own votes and reputation-dependent buttons are merged in last, so the cached body can be shared.

Scaling

In the order the levers get pulled:

1. Cache first. Reads dominate. Stack Overflow's 2016 figures show the shape: about 209 million HTTP requests on a sample day, of which about 66 million were page loads, served by 11 web servers, and the post notes that one web server could carry the load (Craver, 2016). A good cache and tuned queries go a long way before sharding.
2. Take derived work off the database. Search to a search engine, counters to sharded cells, sort keys to precomputed indexes.
3. Read replicas for the listing and thread queries; accept replica lag on non-critical reads.
4. Split hot threads. A single viral thread is a hot key: shard its counters, paginate the tree, collapse deep replies, and cache the rendered top of the thread aggressively.
5. Shard by site or community (one index and one set of tables per community) when the database is the limit. Stack Overflow runs one multi-tenant application for all its Q&A sites, with Stack Overflow's own database on one SQL Server cluster and the other Q&A sites' databases on a second (Craver, 2016).
6. Edge and CDN for anonymous traffic, with short TTLs on counts.

Failure modes

Real implementations

Sources

• Reddit source, archived: _sorts.pyx (hot, controversy, confidence) and comment_tree.py.
• Lemmy, Ranking algorithm and the README.
• E. Miller, How Not To Sort By Average Rating, 2009; and Ranking News Items With Upvotes, 2015 (a derivation of the time-versus-votes trade-off).
• N. Craver, Stack Overflow: The Architecture, 2016 Edition.
• OWASP, Cross Site Scripting Prevention Cheat Sheet.
• Charcoal-SE, SmokeDetector; Discourse, README; Apache Answer, README.