workplace-search

How it is built

Architecture overview

Workplace search answers one query across all the places a company keeps knowledge: documents, chat, tickets, wikis, code, mail, CRM. It differs from web search in two ways. The content lives in dozens of other products that must be connected and kept fresh, and every result must obey the permissions of the source system: a person may find only what they could already open there. The system therefore has an ingestion half (connectors that crawl or receive changes, normalise documents and copy their access lists) and a query half (a permission-aware index that combines keyword and meaning-based retrieval, then re-ranks with signals about people and behaviour). The idea that makes it work is to treat the access list as part of the document and filter on it inside the index, so a leak is impossible by construction, not prevented by a check at the end.

Parts:

• Connectors — pull or receive documents and permissions from each source app — OAuth 2 and OIDC, JWT, Incremental sync cursors, Cursor pagination (no page yet), Message sequence numbers, Write-ahead log, Webhook delivery
• Work control — schedule, throttle and retry the crawls — Token bucket, Exponential backoff with jitter, Durable job queue, Dead-letter queue, Idempotency keys, Circuit breaker, HMAC request signing (no page yet)
• Document processing — text cleaning, tokenization, de-duplication, identity resolution — Unicode text tokenization, Text normalization, SimHash, Entity resolution, Record linkage (no page yet), Union-find, Crypto hashes
• Index — keyword and vector indexes with access lists — Inverted index, Delta encoding, Bit-packing and PFOR, Varint / LEB128, Posting list intersection, Roaring bitmaps, HNSW, d-ary heap, Scalar quantization (no page yet), SIMD programming, embedding-inference (no page yet)
• Ranking — first-stage scoring, fusion and learned re-ranking — BM25 ranking, Hybrid search rank fusion, Top-k selection, Learning to rank, Gradient-boosted trees, Loss functions, Evaluation metrics, AdamW, Reverse-mode autodiff
• Access control — who may see what, on top of single sign-on — Authorization models, CSPRNG (no page yet), Envelope encryption, AEAD ciphers, AES key wrap (no page yet), Mutual TLS
• Operations — backups, failover, deploys, testing — Point-in-time recovery, Copy-on-write snapshots (no page yet), Object storage, Aries recovery, Primary-backup replication, Leases, Health checking, Process supervision, Graceful draining, HDR histogram (no page yet), Deterministic simulation testing, PRNG, Record and replay (no page yet), Fault injection (no page yet), Cooperative scheduling (no page yet), Coverage-guided fuzzing, Code coverage instrumentation, Sanitizers (no page yet), Fast hash functions, Delta debugging (no page yet)

 SaaS apps (docs, chat, tickets, wiki, code)
   │ API + webhooks          ┌────────────── control plane (schedules, rate limits, retries, DLQ) ──────────────┐
   ▼                         │                                                                                  │
 connectors ──docs + ACLs──► processing (clean, tokenize, dedupe, resolve people, embed) ──► index ◄─ ACL sync ─┘
                                                                                          │  keyword + vector + permissions
 user ──SSO──► query service ──query, principal set──► retrieval (BM25 + ANN, filtered) ──► fusion ──► re-ranker ──► results
                  ▲                                                                              ▲ click/dwell logs
                  └──────────────────────────── analytics, evaluation, training ─────────────────┘

How the parts fit together

Connectors

A connector for one source app does four jobs: authenticate, list documents, fetch changes, fetch permissions. Authentication is OAuth 2 and OIDC: an admin grants the product read access once, and the product holds a refresh token per source and per tenant, with scopes as narrow as the source allows. Apps that deliver service credentials or signed assertions use JWT. A first crawl lists everything; after it, the connector stores an opaque cursor from the source (Incremental sync cursors) and asks only for changes since then. Large listings use stable paging (Cursor pagination (no page yet)); event streams use Message sequence numbers to detect missed events. Sources that can push use Webhook delivery: the product exposes an endpoint, verifies the signature (HMAC request signing (no page yet)) and enqueues the work, because a webhook is a hint, never the full truth. The connector's own progress (cursor, last document) is committed through a Write-ahead log in its state store, so a crash resumes mid-crawl rather than restarting.

Every connector outputs the same envelope: a stable document id, source and URL, title, body text, author, times, and an access list of principals (users, groups, "everyone in domain", link-sharing). Normalising permissions into one model is the hardest part of the whole system, because sources differ: Google-style sharing by document, Slack by channel membership, Jira by project role, GitHub by repository.

Work control

Crawls compete for each source's API quota, which is shared by every customer using the same OAuth app. A Token bucket per source and tenant enforces the rate; a 429 response is not an error but a signal to wait for the advertised time, with Exponential backoff with jitter for everything else. Work items (crawl page, fetch document, re-embed) are tasks on a Durable job queue: leased, acknowledged on success, retried on failure, and moved to a Dead-letter queue after repeated failure so one malformed document does not block a million good ones. Tasks carry Idempotency keys so a retry that lands twice indexes once. A Circuit breaker per source stops hammering an unreachable API, and the crawl resumes when the breaker closes. Crawl order matters: recent and frequently viewed documents are fetched first, so a new customer sees useful results in minutes while the long tail fills in for days.

Document processing

Raw text from apps is messy: HTML, markdown, tables, quoted mail threads, boilerplate signatures. Text normalization rewrites it into a canonical form (Unicode normalisation, case folding, whitespace), and Unicode text tokenization splits it into index terms for every language the company uses, including languages without spaces. Documents are cut into passages of a few hundred tokens, because both keyword scoring and embeddings work better on passages than on whole files, and the result view can then jump to the passage.

The same file often exists several times (a slide deck attached in mail, uploaded to a drive, quoted in chat). SimHash gives each document a 64-bit fingerprint whose Hamming distance tracks similarity, so near-duplicates group cheaply; the group shares one ranking slot but keeps each copy's permissions. Content hashes (Crypto hashes) skip re-processing when a document's bytes did not change.

People are the second entity type. "Ana Pereira" in the directory, ana.p in the ticket tool and an email address in the doc comments are one person. Entity resolution blocks candidate pairs, scores them, and merges confirmed matches with Union-find; probabilistic scoring of that kind is the subject of Record linkage (no page yet). The resolved identity powers "documents by this person", boosts for the asker's colleagues, and, most importantly, the mapping from the SSO user to the source-app accounts needed for permissions.

Index

The keyword index is an Inverted index: per term, a sorted list of passage ids with term frequencies. Lists are compressed with Delta encoding of the ids and Bit-packing and PFOR of the gaps, with Varint / LEB128 for the small sparse fields. A multi-term query is a Posting list intersection, skipping through the long list from the short one. The semantic index stores an embedding per passage, produced by embedding-inference (no page yet), in an HNSW graph; its candidate heap is a d-ary heap, distances use SIMD programming, and Scalar quantization (no page yet) cuts vector memory for large tenants at a small recall loss. embedding-inference (no page yet) runs a pretrained embedding model at serving time: it batches the changed passages and the incoming queries, runs the encoder, pools and normalizes the vectors, and caches them, so the vector index and the query path see ready-made unit vectors. The embedding model is never trained here.

Permissions in the index. Each passage carries the ids of the principals who can see it. At query time the caller's principal set (the user plus all their groups) is expanded and the index keeps only passages whose access list intersects it. Both operations are set intersections on integer ids, which Roaring bitmaps do fast: one bitmap per principal of the passages it may read, unioned for the caller and intersected with the term match. The filter is applied before scoring and before counting, so ranking, facets and result counts never reveal what the user cannot open. Group membership changes constantly, so groups are not flattened into each document; document ACLs store group ids, and membership is resolved at query time from a small, fast table.

Ranking

First stage: BM25 ranking over the keyword index and nearest neighbours from the vector index, each returning a few hundred candidates. Hybrid search rank fusion merges them by rank, so no calibration between BM25 scores and cosine similarities is needed, and Top-k selection trims the list. Second stage: Learning to rank re-orders the top candidates with a model trained on query groups, typically Gradient-boosted trees (LambdaMART style) over features such as the keyword and vector scores, document age, source type, author closeness to the asker, the asker's team's recent views, click-through for this query, and title match. Features that depend on the asker are what separates workplace search from web search. Training uses Loss functions built for ordering, and quality is read from Evaluation metrics (NDCG, recall at k) on held-out query sets. Click logs are biased (people click what is shown first), so a model needs position-bias correction or randomised exploration on a small share of traffic; otherwise it learns to reproduce the current order.

Access control

Result filtering handles who sees which document. The rest of the product needs ordinary Authorization models: tenant admins configure connectors, analysts see usage but not content, and every tenant is isolated from every other in storage as well as in queries. Index data holds copies of company text, so it is protected with Envelope encryption: each tenant has its own data key, wrapped by a key service (AES key wrap (no page yet)), encrypting blobs with an authenticated cipher (AEAD ciphers) with nonces from a CSPRNG (no page yet). Deleting a tenant's key makes its data unrecoverable, which is how deletion requests and offboarding are made real in backups. Services call each other over Mutual TLS.

Operations

The relational database holding configuration, cursors and identity uses Primary-backup replication, with Leases and Health checking for failover, Aries recovery after a crash and Point-in-time recovery for operator mistakes (a connector wiped a tenant's cursors). The index is rebuildable from the connectors, but a rebuild takes days, so it is snapshotted (Copy-on-write snapshots (no page yet)) into Object storage. Workers run under Process supervision and finish their current document on Graceful draining. Latency percentiles are recorded as HDR histograms (no page yet), per stage.

Correctness of the crawl scheduler, the ACL sync and the index merge is established with Deterministic simulation testing: a seeded PRNG drives fake source APIs that return errors, reorder events and revoke permissions mid-crawl, with Fault injection (no page yet) and Record and replay (no page yet) reproducing production failures and a Cooperative scheduling (no page yet) runtime making interleavings repeatable. Document parsers (PDF, office formats, HTML) read untrusted bytes and are the likeliest place for a vulnerability, so they are fuzzed (Coverage-guided fuzzing, Code coverage instrumentation, Sanitizers (no page yet), Fast hash functions for the coverage map) and failing inputs are minimised with Delta debugging (no page yet).

Data and control flow

Indexing a changed document:

1. A webhook (or the next scheduled sync) tells the connector that document D changed. The connector fetches D with its current permissions, using the stored cursor to bound the work.
2. Processing normalises text, cuts passages, computes fingerprints, drops unchanged passages by hash, and asks the embedding model for vectors of the changed ones.
3. The index writes new passages and the new access list in one update, then marks old passages deleted. A delete is as important as an insert: a revoked permission must disappear from search quickly.
4. Only after the write is durable does the connector advance its cursor.

A query:

1. The user's browser calls the query service with an SSO session. The service resolves user and groups into a principal set (cached for seconds).
2. Retrieval runs keyword and vector search in parallel, each filtered by the principal set; each returns a few hundred candidates in tens of milliseconds (order of).
3. Fusion and top-k trimming leave about a hundred; the re-ranker scores them with features computed from a feature store; the top ten get snippets and highlights.
4. The response goes back; clicks and dwell time are logged for training and evaluation.

A permission change: the source reports that user U was removed from group G (by webhook or the periodic ACL sync). The group table updates, so every document that lists G stops matching U on the next query. Document-level changes (a file un-shared) update the document's ACL within the sync interval; the lag between the two is the system's published permission staleness.

Scaling

1. Source API quotas are the first limit on freshness, not CPU. Use change feeds and webhooks, prioritise documents by recent use, and spread tenants over separate OAuth apps if a source shares quota per app.
2. Embedding cost. Re-embedding is the most expensive step. Re-embed only changed passages, use a smaller model for the long tail, and batch on GPUs.
3. Index size and query latency. Shard by tenant; large tenants shard by document id with scatter-gather. Quantise vectors, keep hot postings in memory, and cache principal-set bitmaps.
4. Re-ranker latency. Limit candidates to the top hundred and keep features precomputed.
5. Cold start. A new tenant has no click data; start with a global model and personalise as signals arrive.

Failure modes

Real implementations

Glean is the best-known product of this type; its internals are not public, so the table lists open parts only. The alternative is a single-source search embedded in each app: it needs no connector layer and no permission mapping, but users must know where to look, which is the problem this system exists to remove.

Sources

• S. Robertson, H. Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond, Foundations and Trends in Information Retrieval 3(4), 2009.
• G. V. Cormack, C. L. A. Clarke, S. Buettcher, Reciprocal rank fusion outperforms Condorcet and individual rank learning methods, SIGIR 2009.
• Y. A. Malkov, D. A. Yashunin, Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs. paper
• C. J. C. Burges, From RankNet to LambdaRank to LambdaMART: An Overview, Microsoft Research TR-2008-109. report
• RFC 6749, The OAuth 2.0 Authorization Framework. rfc
• Onyx documentation: docs.onyx.app.