← Back
Home › Systems › Web, content and learning

docs-platform

Turns Markdown pages and API descriptions in Git into a fast, searchable, versioned documentation website for product and developer docs.

Mintlify, GitBook

Open the docs-platform pagedocs-platform Citation groundingagentic-ai: Attaching answer spans to retrieved source passages and verifying each claim is supportedCDN and edgecloud: CDNs cache content in hundreds of PoPs near users, cutting latency and egress bills; edge compute runs small code there. Bandwidth price spreads 10x between providers.›Idl code generationcompilers: Generate typed client/server stubs and codecs from an interface definitionTemplate enginescompilers: Compile templates with loops, partials and auto-escaping into render functionsHdr histogramdata-structures: Log-linear bucketed histograms with bounded relative error for latency percentilesDeterministic simulationtestingdistributed: Run the whole system in one thread under a seeded scheduler with simulated network, disk and clock, so every failure replays from its seed.›Primary-backup replicationdistributed: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›Three-way mergedistributed: Merge two descendants against their common ancestor: take whichever side changed, report a conflict only where both changed the same spot differently.›Code splitting lazy loadinggraphics: Split bundles at dynamic imports, load chunks on demand, preload critical pathsMarkdown parsinggraphics: CommonMark two-phase parsing: block structure, then inline delimiter runsRich text document modelgraphics: Schema-constrained document trees with marks, positions and transactional steps (ProseMirror-style)Ssr hydrationgraphics: Attach client state and listeners to server-rendered HTML without re-rendering; mismatch handlingSyntax highlightinggraphics: TextMate grammars or parser-based highlighting into token spansProcess supervisionhardware-os: A parent process starts services, reaps their exits and restarts them under a delay and burst-limit policy, then stops them with SIGTERM before SIGKILL.›Embedding inferencemachine-learning: Serve a pretrained embedding model: tokenize, batch requests, run the encoder, pool and normalize vectors, cache them; no trainingText chunkingmachine-learning: Cut documents into token-bounded pieces along structure boundaries, keeping offsets and section paths, so each piece embeds well and can be cited.›Webhook deliverynetworking: Reliable server-to-server HTTP callbacks: signed POSTs, 2xx-means-done, retries over days with jitter, endpoint disabling, SSRF guards and idempotent receivers.›Hybrid search and rank fusionsearch-sort: Merge a keyword (BM25) result list and a vector (ANN) list into one ranking, by ranks (RRF) or by normalized scores (convex combination).›Myers diffsearch-sort: Greedy O(ND) shortest edit script between two sequences, the default of git and GNU diff; fast when the inputs are similar.›Trigram indexsearch-sort: Trigram posting lists accelerating regex and substring searchAcme protocolsecurity: Automated TLS certificate issuance and renewal via ACME HTTP-01/DNS-01 challengesCoverage-guided fuzzingsecurity: Mutate inputs, run an instrumented target, and keep any input that reaches a new code edge; feedback turns blind guessing into incremental search.›JSON Schema validationsecurity: Compile a JSON Schema once, then check every untrusted document against it, within hard size and depth limits, and report failures as JSON Pointers.›Mutual TLSsecurity: TLS where the client also proves a certificate key, so every connection carries a verified, rotatable machine identity.›OAuth 2 and OpenID Connectsecurity: Delegated authorization (OAuth 2) and login on top of it (OIDC); use authorization code + PKCE for every user-facing client.›Git object modelstorage: Content-addressed blobs, trees, commits and refs; packfiles with delta compression; worktreesInverted indexstorage: Map each term to a sorted, compressed list of the documents containing it; the core of full-text search and other fast set queries.›Point-in-time recoverystorage: Periodic base backups plus a continuously archived change log let you rebuild the state as of any instant, then branch from there.›Cache invalidationsystems: Make writes remove cached copies by key, group tag or prefix, broadcast the removal to every node, and guard refills with tokens so stale values never return.›Graceful drainingsystems: On a termination signal, stop taking new work, finish in-flight work until a deadline, tell clients to reconnect elsewhere, then exit; clients resume from a token.›Text classificationmachine-learning: Assign one or more labels to a text; climb a ladder from regexes to linear models on n-grams, embeddings, fine-tuned encoders and LLM prompts, only as far as the data and budget demand.›Cross-encoder rerankingsearch-sort: Re-score the top-k candidates of a cheap first stage with a model that reads query and candidate together; more accurate than separate embeddings, but one model call per pair.›Fuzzy string matchingsearch-sort: Edit distance, Jaro-Winkler and trigram indexes for approximate name and address matchingTLS 1.3security: The standard encrypted, authenticated transport; one round trip, only forward-secret AEAD suites, and little left to misconfigure.›Protobuf wire formatnetworking: Protocol Buffers wire format: tags, varints, length-delimited fields and schema evolutionName resolutioncompilers: Bind every use of a name to its declaration with a scope stack, handling shadowing, hoisting and closure capture in one AST walk.›Html sanitizationsecurity: Allowlist-based HTML sanitizing of untrusted markupHTTP caching semanticsnetworking: The rules (RFC 9111) that decide whether a stored response is still fresh, and how a cheap conditional request revalidates it when it is not.›Recursive-descent parsingcompilers: Hand-written top-down parser with one function per grammar rule; simple, fast, with full control over error messages and recovery.›Cooperative schedulingconcurrency: Green threads/coroutines yielding at await points on a user-space schedulerFault injectiondistributed: Inject latency, errors, kills and partitions via proxies, netem or hooksRecord replaydistributed: Record every nondeterministic input (time, randomness, I/O, model and tool outputs, per-tick player input) and re-execute deterministically for debugging, replays and state rebuild+ 47 moreCalibrationmachine-learning: Make predicted probabilities match observed frequencies: measure with ECE, fix post hoc with temperature, Platt or isotonic scaling on held-out data.›Linear and logistic regressionmachine-learning: Fit a weighted sum of features to a number (least squares) or a probability (logistic); the baseline every model must beat.›Transformer blockml-training: The decoder-only pre-norm block (attention plus MLP on a residual stream) behind every modern LLM, with exact shapes, parameter, FLOP and KV-cache counts.›AEAD (AES-GCM,ChaCha20-Poly1305)security: Authenticated encryption that encrypts and detects tampering in one call; the only kind of symmetric encryption you should use.›ECDH key exchange (X25519,P-256)security: Each side multiplies the other's public curve point by its own scalar and both get the same point; always validate the peer key and run the result through a KDF.›HKDF key derivation(extract-and-expand)security: HMAC-based two-step key derivation: extract a uniform pseudorandom key from any secret, then expand it into as many context-bound keys as you need.›Transparent interceptionnetworking: Redirect traffic to a local proxy (iptables TPROXY/REDIRECT, eBPF) without app changesSyscall interceptionsecurity: Intercept syscalls via ptrace, seccomp user notify or a user-space kernel+ 22 moreLoss functionsmachine-learning: The scalar a model minimizes: cross-entropy for classes and tokens, MSE or Huber for values, each with a stable form and simple gradient.›L-BFGSnumerics: Quasi-Newton minimizer that approximates the inverse Hessian from the last m gradient differences via the two-loop recursion; O(m·n) memory and work per step.›Constant time codesecurity: Writing secret-dependent code without branches or memory lookups on secrets; constant-time compare and selectEbpfhardware-os: Verified in-kernel programs on hooks (kprobes, XDP, tc) with mapsPtrace process controlhardware-os: Ptrace: attach, read and write registers and memory, single-step, intercept signalsSeccomp bpfsecurity: Syscall filtering with seccomp BPF programs+ 6 more+ 1 more

How it is built

Architecture overview

A documentation platform turns a repository of Markdown pages and an API description into a fast, searchable, versioned website, and keeps it correct while many people edit it. The writer's source of truth is text in Git (or a web editor that commits to Git). A publish pipeline parses the pages, validates them, renders them with components, generates API reference pages from an OpenAPI file, builds a search index, and writes the result to a CDN. At read time the site is mostly static files; the dynamic parts are search, an assistant that answers questions from the docs, and access control for private docs.

The idea that makes it work is one content model feeding several readers. The same parsed pages become an HTML page for people, a clean Markdown page and an llms.txt index for AI agents, chunks for the search index and for retrieval, and entries in a sitemap. Mintlify and GitBook, the example products, both publish docs for agents next to docs for people (Mintlify documentation lists llms.txt, a Markdown export and MCP servers; GitBook documentation lists LLM-ready docs and MCP servers for published docs). The value is mostly around the renderer: review, previews, link checking, redirects and fast delivery.

This page is the platform. The build mechanics of a general site are in Static site generator; collaborative editing is Real-time collaborative document editing; the reader's help center variant is Help center and knowledge base; retrieval for answers is Retrieval-augmented generation (RAG); a company's service catalogue is Internal developer portal and service catalog; languages are Translation management; and review of a content change follows Code review system.

Parts:

 writers ── web editor / Git push ──► repo (pages, openapi.json, config)
                                         │ webhook "push"
                                         ▼
                          build worker: parse ─► validate (links, schema) ─► render HTML
                               │                    │                         │
                               │ OpenAPI ─► endpoint pages                    ├─► page chunks ─► embeddings
                               │ index (words) ──────────────────────────────►│        │
                               ▼                                              ▼        ▼
                         artifact store (HTML, JS, .md, llms.txt, search index)   vector index
                               │ publish: atomic pointer switch + purge          │
                               ▼                                                 │
 readers ── browser ◄── CDN ◄── origin ◄── edge/app: auth, redirects, search API ┘
 AI agents ◄── /llms.txt, .md pages, MCP         assistant: retrieve ─► LLM ─► answer + citations

How the parts fit together

Source and editing

Git is the right source of truth for docs because it brings branches, review, history and blame for free. Platforms add a layer on top: Mintlify's documentation describes a repository setup, a CLI for local preview, a web editor and CI checks; GitBook's describes GitHub and GitLab sync, change requests with merge rules, comments and live edits. Two edits to the same page are reconciled with a Three-way merge: take whichever side changed each region against the common ancestor, and report a conflict only where both changed the same lines. The line-based diff shown in review comes from Myers diff. A web editor that non-programmers use works on a structured document tree with marks and positions (rich-text-document-model, no page yet), converts it to Markdown or MDX on save and commits it to a branch. If several people type in the same page at once, see Real-time collaborative document editing. Git's content-addressed trees and commits (git-object-model, no page yet) make "what changed since the last build" cheap to compute.

The contract between this part and the build: a commit is a complete, self-consistent snapshot of pages, assets, navigation config and API spec. The build never reads from the editor, only from Git.

Content parser

Pages are Markdown, usually extended with components. CommonMark defines parsing in two phases, block structure first (headings, lists, quotes, code blocks) and then inline structure inside each block (emphasis, links, code spans) (CommonMark specification), so a conforming parser is deterministic and the same page renders the same everywhere (markdown-parsing, no page yet). Component extensions come in two main flavours. MDX lets you use JSX in Markdown, so pages can import and embed components (MDX documentation). Markdoc, designed at Stripe for its documentation, adds tags and annotations to Markdown but is by design not a templating language that mixes arbitrary code with content (Markdoc documentation). The trade is power against safety: MDX allows anything React can do, so a page can break the build; a tag-only format can be validated against a schema first, which suits many non-engineer authors.

Both give a syntax tree (headings, text, tags with attributes) that the rest of the pipeline walks. From the tree come the table of contents, the plain text for search, the chunks for retrieval and the Markdown export. Code blocks are highlighted at build time into styled spans so the browser runs no highlighter; Shiki, for example, is based on TextMate grammars (syntax-highlighting, no page yet; Shiki README). Snippets and variables are resolved here (template-engines, no page yet).

API reference generator

API docs are generated, not written. An OpenAPI description is a JSON or YAML document of paths, operations, parameters and schemas; text fields may contain CommonMark (OpenAPI Specification), so descriptions use the same parser as pages. The generator validates the file, resolves references, and emits one page per operation with parameters, response schemas and examples. Schemas are JSON Schema validation documents, which can also check request examples against their schema so published examples are not wrong. Typed code samples in several languages are derived from the same description (idl-code-generation, no page yet). Mintlify's documentation describes endpoint pages and an API playground generated from specifications; GitBook's has a "Document an API" section. The contract: the OpenAPI file is a build input and never edited by the platform; a bad file fails the build with a message pointing at the line.

Site renderer

The renderer turns the tree into HTML with the site's layout and components. Docs are read-heavy and content-stable, so the default is to render at build time and ship static files, with a small amount of JavaScript for interactive pieces (tabs, code groups, the search box, the playground). Where components need state, the server-rendered HTML is "hydrated": the client attaches listeners and state to markup that already exists instead of rendering from scratch (React's hydrateRoot does this for HTML produced on a server), and a mismatch between server and client output is a bug class of its own (React documentation) (ssr-hydration, no page yet). Heavy components such as the API playground are split into bundles loaded on demand (code-splitting-lazy-loading, no page yet). Docusaurus, the best-known open-source docs framework, describes itself as a static-site generator that builds a single-page application with client-side navigation using React (Docusaurus documentation).

Reader search must feel instant and handle code identifiers, which ordinary word tokenizers split badly. A static, in-browser index is the cheapest: Pagefind runs after the site is built and writes a static search bundle into the output, so no search server is hosted, and it aims to use as little of the reader's bandwidth as possible (Pagefind documentation). Underneath is an Inverted index over words after Unicode text tokenization for search, with ranking by term statistics. Substring search for identifiers and error messages needs a character-level structure such as a trigram index (trigram-index, no page yet).

Keyword search misses questions phrased in other words. The semantic side embeds chunks (Text chunking, Embeddings) and finds nearest vectors; the two ranked lists are merged by Hybrid search and rank fusion, which needs no score calibration if rank-based. The indexer reads the same parsed tree as the renderer, so every hit points to a stable section anchor.

Assistant

The assistant answers questions in natural language from the docs. It is Retrieval-augmented generation (RAG): retrieve the best chunks for the question with the hybrid search above, give them to a language model with an instruction to answer only from them, and show which pages were used. Chunking along headings, with the section path stored as metadata, keeps passages coherent and citable. Turning chunks and questions into vectors is embedding-inference (no page yet): one embedding model, run in batches over changed chunks at publish time and once per question at query time, so both land in the same vector space; a model change means re-embedding every chunk, so the model version is stored with each vector. Grounding means checking that each claim in the answer is supported by a retrieved passage and linking to it (citation-grounding, no page yet). The platform gives the docs to external agents too: the proposed /llms.txt file is a Markdown index of a site's pages meant to be read by language models (llmstxt.org), and both example products publish MCP servers for their docs.

Delivery and domains

Built artifacts go to an artifact store and are published by switching one pointer, so readers never see half a build. A CDN and edge caches pages near readers. After a publish, the old pages must stop being served: either use fingerprinted asset names with short-lived HTML, or purge by key, tag or prefix (Cache invalidation). Custom domains need a TLS certificate per customer; automated issuance and renewal is the ACME protocol (acme-protocol, no page yet). Redirects, a feature both example products document, are a table consulted at the edge so that a renamed page keeps its old URL alive.

Identity and integrations

Private docs sit behind login. Readers authenticate through OAuth 2 and OpenID Connect against the customer's identity provider; the edge checks the session before serving a page or the search API, so content under access control never lives in a public static index. Git pushes and merges reach the platform as Webhook delivery events that trigger a build; build results go back as commit statuses, and outbound webhooks or chat messages announce publishes.

Operations

The build workers are a queue of short jobs; the app tier is stateless apart from the metadata database and the stores. Process supervision restarts crashed workers; Graceful draining lets a deploy finish builds in flight. The metadata store has Primary-backup replication and Point-in-time recovery; content itself is recoverable from Git. The parser and OpenAPI loader read untrusted input, so they get Coverage-guided fuzzing and hard limits on size and depth. Deterministic simulation testing suits the publish pointer switch and purge ordering. Latency percentiles use hdr-histogram (no page yet). Internal service calls can use Mutual TLS.

Data and control flow

A writer changes a page and publishes:

  1. The writer edits in the web editor or in their own editor and pushes a branch. The platform builds a preview from that branch with its own URL.
  2. A push webhook arrives; the control plane enqueues a build for the commit, checking out the tree.
  3. The worker parses every page. Unchanged pages (same content hash) are reused from cache; the others are parsed, validated (broken links, unknown components, invalid OpenAPI) and rendered.
  4. The worker writes HTML, bundles, Markdown copies, llms.txt, the search index and the retrieval chunks; embeddings are computed only for chunks whose text changed.
  5. Reviewers see the preview; checks report problems on the pull request.
  6. On merge to the default branch, the same build runs for production; the publish step switches the pointer and purges the changed URLs. Time from merge to live is the build plus purge; mostly parse and render time on a large site.

A reader finds an answer:

  1. The browser requests a page; the CDN returns it from cache, or fetches from origin after checking access.
  2. The reader types in the search box; the client queries the static index or the search API and shows results as they type.
  3. If the reader asks the assistant, the API embeds the question, runs hybrid retrieval, builds the prompt from the best chunks, calls the model, and streams the answer with links to the source sections.
  4. Searches with no results and assistant questions without good sources are logged and shown to writers as gaps.

Scaling

In the order the limits appear:

  1. Build time on large sites. Thousands of pages, images and an API spec. Levers: content-hash caching of parse results, building only what changed, parallel workers, incremental search and embedding updates, preview builds on branches only for changed pages.
  2. Many builds at once. Levers: a queue with per-customer fairness, autoscaled ephemeral workers.
  3. Cache hit ratio and publish latency. Purging too broadly makes every publish a cold start. Levers: fingerprinted assets, surrogate keys per page and per navigation change, short TTL only on HTML shells.
  4. Search and retrieval. The index grows with content and with history if old versions are kept. Levers: a static index for small sites, a hosted or sharded index for large ones, separate indexes per version and language, caching frequent questions.
  5. Assistant cost and latency. Each answer is a model call plus retrieval. Levers: retrieve fewer, better chunks, cache answers to common questions, cheaper models for simple lookups, rate limits per site.
  6. Versions and languages. Levers: share unchanged pages across versions, redirect old versions, a translation workflow (Translation management).

Failure modes

FailureSymptomCauseFix
Broken build blocks publishsite stays on an old versionone invalid page, schema or component breaks the whole buildfail with file and line; allow partial builds with the bad page skipped and flagged
Stale pages after publishreaders see old contentpurge missed a URL or an edge kept the old HTMLsurrogate keys per page; versioned asset names; verify after publish
Broken links404s inside docspages renamed or movedlink check in CI; redirect table updated automatically on rename
Hydration mismatchflicker or console errors on loadserver output differs from client output (time, locale, browser-only data)render the same on both sides; defer client-only parts
Search misses code termsexact identifier gives no hittokenizer splits on punctuationindex identifiers whole and split; trigram fallback
Assistant invents an answerconfident text with no support in docsretrieval returned irrelevant chunks, or the model ignored themrequire citations, refuse when sources are weak, check support of each claim
Private content leaksprivate page text appears in search or assistantshared index without access filterfilter at query time by the reader's rights; separate indexes per audience
Certificate expirybrowser warning on a custom domainrenewal failed (DNS or challenge)monitor expiry, alert before the deadline, retry renewal

Real implementations

namewhat it isrepolicencenotes
DocusaurusReact-based static docs frameworkfacebook/docusaurusMITbuilds a React single-page application per its docs
Starlightdocumentation website framework for Astrowithastro/starlightMITbuilt on Astro
MkDocsstatic docs generator from Markdownmkdocs/mkdocsBSD-2-Clauseproject documentation from Markdown
Read the Docssource code that powers readthedocs.orgreadthedocs/readthedocs.orgMIThosted docs service
GitBookdocs platform with GitHub and GitLab sync, change requests and AI featuresGitbookIO/gitbookGPL-3.0the open-source frontend for GitBook doc sites per its description; the product is at gitbook.com
Mintlifyhosted docs platform with CLI, web editor, API playground, assistant——proprietary service; docs at mintlify.com/docs; starter template mintlify/starter is MIT
MarkdocMarkdown-based format and framework from Stripemarkdoc/markdocMITparser and renderer building block
MDXMarkdown with JSXmdx-js/mdxMITparser building block
Shikisyntax highlighter based on TextMate grammarsshikijs/shikiMITbuild-time highlighting
Pagefindstatic search libraryCloudCannon/pagefindMITindex built after the site build

The alternative: hand-built docs inside the product's own web app or wiki. It is quickest to start and gives complete control, but loses versioned, reviewable source, generated API pages and agent-ready output, and each of these has to be built again. A hosted platform wins when many non-engineers write; a static framework wins when engineers own docs and want no vendor.

Sources

Similar systems

Share the most building blocks with this one.

knowledge-base 42 %translation-management 37 %static-site-generator 34 %developer-portal 34 %code-review 32 %