knowledge-base

How it is built

Architecture overview

A knowledge base is a set of articles that customers and staff search and read instead of asking a person. Someone writes an article in an editor; the system stores it as a structured document with revisions; a publisher renders it to pages served through a CDN; and a search service lets readers find it by typing a question. Modern help centers add a chat or "ask" box that answers in a few sentences and shows the articles it used. The one idea that makes it work: the article revision is the single source of truth, and everything readers touch (HTML page, keyword index, vector index, answer snippets) is a derived copy rebuilt from the revision stream. That keeps edits cheap, lets search be swapped without touching content, and lets visibility rules be applied at one place. Zendesk Guide and Document360 are the well-known hosted products; Docusaurus, BookStack, Wiki.js and MediaWiki cover open-source authoring, and Lucene-based or Meilisearch-style engines cover search.

Parts:

• Authoring — structured editor model, safe HTML, revision history and diff — Rich-text document model (no page yet), HTML sanitization (no page yet), Myers diff
• Publishing and delivery — render pages and media, serve from the edge — CDN and edge, HTTP caching semantics, Image codecs (no page yet)
• Keyword search — analyse text and rank matches — Inverted index, BM25 ranking, Stemming (no page yet), Query expansion (no page yet), Language identification
• Semantic search — find articles by meaning, merge with keywords, reorder — embedding-inference (no page yet), Text chunking, Hybrid search and rank fusion, Cross-encoder reranking
• Answer layer — compose an answer from retrieved passages and show where it came from — Citation grounding (no page yet), system page Retrieval-augmented generation (RAG)
• Access — public, customer-only and staff-only content, single sign-on — Authorization models (RBAC, ABAC, ReBAC), OAuth 2 and OpenID Connect
• Search analytics — what people searched, whether it worked — Sessionization (no page yet)
• Operations — keeps the content, the indexes and the serving path safe and testable — Mutual TLS, Point-in-time recovery, Primary-backup replication, Process supervision, Graceful draining, hdr-histogram (no page yet), Deterministic simulation testing, Coverage-guided fuzzing

 editor ──► article revision (structured doc, locale, audience) ──► revision log
                 │ sanitise, diff                                     │
                 ▼                                                    ▼
          publisher ──► HTML + images ──► origin ──► CDN ──► reader's browser
                 │                                              │ search box / ask box
                 ├──► analyser (language, stems) ──► inverted index ──┐
                 └──► chunker ──► embedder ──► vector index ──────────┤
                                                                      ▼
                       query ► language id ► expansion ► BM25 leg ──┐
                                           └──► vector leg ───────► rank fusion ► rerank ► results
                                                                                      └► LLM answer + citations
 search events ──► sessionize ──► zero-result and click reports ──► authors

How the parts fit together

Authoring

Articles are written in a visual editor, imported from Markdown, or pasted from other tools. A Rich-text document model (no page yet) stores the content as a tree of typed blocks (heading, list, code, callout, image) with inline marks, instead of an HTML string. A tree is the same for every consumer: the page renderer, the chunker, the plain-text extractor for search, a translation tool. HTML is then an output format, not the storage format.

Because articles can still arrive as HTML (imports, an API, an "edit source" mode), everything that comes in passes HTML sanitization (no page yet): allow-listed tags and attributes, no scripts or event handlers, safe URL schemes. Zendesk's Help Center API documents this directly: an article body is HTML, and "unsafe tags and attributes may be removed before display" (source: Zendesk developer docs, Articles). DOMPurify is a browser-side example of the pattern; OWASP's cross-site-scripting guidance covers output encoding.

Every save creates a revision. Editors need to see what changed, reviewers need to approve a change, and a bad edit needs a one-click revert. Myers diff compares two revisions as a shortest edit script; for structured documents it runs on the sequence of blocks first and on words inside changed blocks second, which gives readable output. The contract between authoring and the rest: publishing emits an immutable revision id. All downstream jobs key their work on that id, so they can be retried or re-run for any revision without surprises.

Multilingual help centers keep one article per locale and mark each translation as draft or published; Zendesk's article object exposes a draft flag per locale translation (source: Zendesk developer docs, Articles). The locale becomes part of the key in every derived store.

Publishing and delivery

Reading is almost entirely GET requests for pages that change rarely, so the read path should not touch the database. The publisher renders a revision to HTML (and copies images), and CDN and edge nodes serve it close to the reader. HTTP caching semantics is the contract between publisher and edge: responses carry Cache-Control with a lifetime and an ETag, so most requests are answered at the edge and a stale copy is revalidated with a conditional request that returns 304 without a body (RFC 9111). Publishing a new revision must make the old page stop being served: either a short lifetime, or purging the URL at the edge, or versioned URLs for assets (a hash in the file name) with a short lifetime only on the HTML.

Media is usually the heaviest part, so uploads go through Image codecs (no page yet): resize to a few widths, re-encode to a modern format with a fallback, strip metadata, and let srcset pick. Pages for private articles (below) must be private or uncached, since a shared cache must not hand one customer's page to another.

Keyword search

Lexical search is still the backbone: customers type error messages, product names and exact phrases. An Inverted index maps each term to the articles that contain it. The analyser in front of it decides what a term is: lower-casing and Unicode-aware splitting, then Stemming (no page yet) so "printing" and "printed" meet, and a language-specific stop-word list. Which analyser to use depends on the article's locale; for queries whose language is unknown, Language identification picks it, and short queries are the hard case: when the detector says unknown, search with the account's default language, or with a language-neutral analyser, rather than guess.

BM25 ranking orders matches: rare terms count more than common ones, repeated terms saturate, and long articles are normalised (the probabilistic relevance framework of Robertson and Zaragoza). Fields are weighted, title above body above comments, because a title match is nearly always what the reader wants. Query expansion (no page yet) adds synonyms and product aliases ("log in" and "sign in", an old product name), maintained as a small curated list by the support team. The cost: a wrong synonym hurts every query that contains it.

Semantic search

Keyword search fails when the reader's words differ from the author's ("my phone won't charge" against "battery troubleshooting"). embedding-inference (no page yet) maps a query and a passage to vectors whose distance reflects meaning: a pretrained embedding model is run at serving time on every chunk when a revision is published and on every reader question, with batching and a cache for repeated questions, and no training in this system. Articles are too long to embed as one vector, so Text chunking cuts them along headings and paragraphs into passages of a few hundred tokens, keeping the article id, the heading path and the offset so a hit can link to the exact section. The vector index is rebuilt per revision: delete the chunks of the old revision, insert the new ones.

The two legs are merged by Hybrid search and rank fusion. Reciprocal rank fusion adds 1 / (k + rank) over the lists, with k = 60 in the original paper (source: Cormack, Clarke and Buettcher, SIGIR 2009), and needs no score normalisation, which matters because BM25 scores and cosine similarities are on different scales. Elastic recommends RRF for hybrid search in its own documentation. The merged top 50 or so go to Cross-encoder reranking, a model that reads query and passage together and re-scores each pair, as in the BERT passage re-ranking work of Nogueira and Cho. A cross-encoder is too slow for the whole corpus, which is why it only sees the short list. The contract: each stage hands the next a ranked list of chunk ids with article ids, and the final step collapses chunks to articles for display.

Answer layer

Instead of ten links, an "ask" box can write a short answer. This is the Retrieval-augmented generation (RAG) pattern: retrieve passages with the machinery above, give them to a language model, and ask it to answer only from them (Lewis et al. introduced the term). Citation grounding (no page yet) is what makes it safe for support: every statement in the answer points to a passage id, the UI shows those articles, and an answer with no supporting passage is replaced by "I could not find this; contact support". Retrieval quality limits answer quality, so improvements to search help here first. Document360 advertises chatbots over knowledge-base sources (source: its docs home).

Access

Not every article is for everyone: some are public, some for logged-in customers, some for agents only, some for one customer's plan. Authorization models (RBAC, ABAC, ReBAC) describe the choices: roles ("agent", "admin"), attributes (plan, region), or relationships (the customer belongs to organisation X). Most help centers start with role plus audience tags per article: an article has an audience; a reader has roles and groups; the check is a set intersection. Zendesk documents a roles model where admins create and edit articles, other agents need management permissions, and end users cannot edit (source: Zendesk developer docs, Articles).

The important rule: filter before ranking, in the index query, not after. If a private article is retrieved and then hidden, its title can still leak through snippets, suggestions, counts or the AI answer. Each chunk in the vector index carries the same audience tags as its article, and every search runs with the reader's allowed audiences as a filter. Login to the customer area is delegated with OAuth 2 and OpenID Connect: the company's identity provider asserts who the customer is, and the help center reads the group claims.

Search analytics

Sessionization (no page yet) groups a reader's events (query, result click, article view, "was this helpful", ticket created) into sessions by inactivity gap. From the sessions you get the reports authors need: queries with no results, queries with results but no click, articles viewed and then followed by a ticket. Those are the gaps to write or fix. It is also the only honest way to judge search changes: compare hybrid against lexical on click and ticket outcomes, not on offline guesses.

Operations

The revision log and article store are the authors' work, so they are replicated with Primary-backup replication and covered by Point-in-time recovery: an editor who overwrites a whole category or a bad import can be rolled back to the minute before, and the publisher re-renders pages and rebuilds the keyword and vector entries from the restored revisions. The indexes and the CDN are derived data and are rebuilt, not backed up.

The publisher, the indexers, the search service and the answer service run under Process supervision. A deploy uses Graceful draining: the search and answer workers finish the queries in flight (an answer can take seconds to stream) and a publisher finishes the revision it is rendering before it exits, so a reader never gets half an answer and an article is never left half-indexed. The internal calls from the origin to search, from search to the embedding and reranking models, and from the answer layer to the retriever go over Mutual TLS, so internal-only and staff-only articles cannot be fetched by anything outside the mesh.

Search quality is useless if it is slow, so query latency is recorded as an hdr-histogram (no page yet) per stage (keyword leg, vector leg, rerank, answer start), and the tail percentiles say which stage is eating the budget.

The publish pipeline (revision saved, page rendered, indexes updated, CDN purged) is tested with Deterministic simulation testing: a simulated clock and queue reorder and repeat these steps and crash between them, and the test checks that readers never see a page whose index entry is missing or an unpublished article in results. The sanitiser, the rich-text importer and the query parser all read bytes from strangers, so Coverage-guided fuzzing runs them on mutated articles and pasted HTML and keeps every input that reaches new code.

Data and control flow

Publishing an article:

1. An author saves; the editor writes a new revision (document tree, locale, audience, author) and the diff against the previous one.
2. Sanitisation runs on any HTML-sourced content. The revision gets an id and is appended to the revision log.
3. On "publish", the publisher renders HTML for the locale, processes new images, writes to the origin and purges or version-stamps the edge copy.
4. A job per revision id analyses the text and updates the inverted index; another chunks, embeds and replaces the vector chunks. Embedding is the slow, batchable step.
5. The article becomes searchable when both indexes carry the new revision. Until then the old revision stays visible; the switch is by revision id, not by deleting first.

A reader asks a question:

1. The browser sends the query with the session and, if logged in, the identity; the edge passes it on (search results are not cached at the edge unless the query and audience are identical).
2. The service detects the language, expands synonyms and builds the audience filter.
3. Both legs run in parallel: BM25 over the inverted index and nearest-neighbour over the vector index. Both apply the filter.
4. Fusion merges the lists; the cross-encoder re-scores the short list; chunks collapse to articles.
5. Results are returned with snippets from the best chunk. For the ask box, the top passages go to the language model with an instruction to cite; the answer is checked for citations before display.
6. The query, results and clicks are logged for the analytics sessions. Reranking and generation usually dominate latency, so the page can show results first and the answer after.

Scaling

Pull the levers in this order:

1. Read traffic. Cache at the edge and keep pages static; this scales to very large read volumes with one origin. Lever: longer lifetimes plus purge on publish.
2. Search latency. Lever: filter early, cap candidate counts per leg, run the legs in parallel, rerank only the short list, cache popular queries.
3. Index size. Chunks multiply entries several times over. Lever: larger chunks, smaller vectors (quantisation), index per locale, drop retired articles.
4. Re-embedding cost. A new embedding model means re-embedding everything. Lever: version the vector index, build the new one beside the old, switch with a flag, and keep the revision stream as the source to rebuild from.
5. Authoring concurrency. Lock per article or merge by block; the revision log keeps it recoverable.

Failure modes

Real implementations

The alternative: a docs-as-code site (Markdown in git, static build, a client-side search index) is cheaper and fine for product documentation owned by engineers. A hosted help center earns its cost when many non-technical authors, per-audience visibility, translations and analytics are needed. Either way the read path is static pages at the edge and the search leg is the part worth engineering.

Sources

• S. Robertson, H. Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond, Foundations and Trends in Information Retrieval, 2009. doi
• G. Cormack, C. Clarke, S. Buettcher, Reciprocal rank fusion outperforms Condorcet and individual rank learning methods, SIGIR 2009. doi
• R. Nogueira, K. Cho, Passage Re-ranking with BERT, 2019. paper
• P. Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020. paper
• E. Myers, An O(ND) difference algorithm and its variations, Algorithmica, 1986. doi
• RFC 9111, HTTP Caching. rfc
• OWASP, Cross Site Scripting Prevention Cheat Sheet. page
• Zendesk developer docs, Articles (Help Center API). docs
• Elastic docs, Hybrid search. docs
• Document360 documentation. docs