← Back
Home › Systems › Sales, marketing and customer service

experience-management

enterprise survey and experience management

Qualtrics, Medallia, Culture Amp

Open the experience-management pageexperience-management Pratt parsingcompilers: Table-driven expression parsing with one loop and a numeric binding power per operator; the compact core of precedence handling in hand-written parsers.›Hdr histogramdata-structures: Log-linear bucketed histograms with bounded relative error for latency percentilesDeterministic simulationtestingdistributed: Run the whole system in one thread under a seeded scheduler with simulated network, disk and clock, so every failure replays from its seed.›Primary-backup replicationdistributed: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›Process supervisionhardware-os: A parent process starts services, reaps their exits and restarts them under a delay and burst-limit policy, then stops them with SIGTERM before SIGKILL.›Embedding inferencemachine-learning: Serve a pretrained embedding model: tokenize, batch requests, run the encoder, pool and normalize vectors, cache them; no trainingGeneralized linear models(GLMs)machine-learning: Linear model for counts, rates, positive amounts and proportions: a linear score pushed through a link, fitted by iteratively reweighted least squares.›Shapley values treeshapmachine-learning: TreeSHAP and Shapley attributions for per-prediction explanations and reason codesText classificationmachine-learning: Assign one or more labels to a text; climb a ladder from regexes to linear models on n-grams, embeddings, fine-tuned encoders and LLM prompts, only as far as the data and budget demand.›Email delivery smtpnetworking: Outbound SMTP: queues, MX lookup, retries, bounces, feedback loops and IP warm-upOpen and click trackingnetworking: Tracking pixels and signed redirect links for engagement eventsMarkov chain monte carlonumerics: Metropolis-Hastings and Gibbs sampling: proposals, burn-in, thinning, convergenceCoverage-guided fuzzingsecurity: Mutate inputs, run an instrumented target, and keep any input that reaches a new code edge; feedback turns blind guessing into incremental search.›K anonymitysecurity: Minimum group-size thresholds and suppression so aggregates cannot single out a personMutual TLSsecurity: TLS where the client also proves a certificate key, so every connection carries a verified, rotatable machine identity.›Row-level security(multi-tenant isolation in the…security: Append a policy predicate to every query inside the database, so a forgotten WHERE tenant_id = ? in application code cannot leak another tenant's rows.›Samlsecurity: SAML 2.0 assertions, XML signatures, SP/IdP flows and canonicalization pitfallsScim provisioningsecurity: SCIM 2.0 user/group provisioning API: create, patch, deprovision, filtersConjoint choice designstatistics: Build balanced choice sets (orthogonal or D-optimal designs) and estimate utilities by multinomial logit or hierarchical BayesHypothesis testingstatistics: Pick a test statistic, compare it to its distribution under "no effect", read off a p-value, and control error rates across one test or many.›Randomization blockedstratifiedstatistics: Permuted-block, stratified and minimization randomization with concealed allocationSurvey weighting rakingstatistics: Raking (iterative proportional fitting) of survey weights to population marginsClosure tablestorage: Storing trees and hierarchies in relational tables (closure table, materialized path) for fast subtree queriesColumnar storagestorage: Store each column contiguously so analytic scans read only the columns they need, compress well and run as tight SIMD loops.›Point-in-time recoverystorage: Periodic base backups plus a continuously archived change log let you rebuild the state as of any instant, then branch from there.›Rollup aggregationstorage: Fold raw events into one mergeable partial aggregate per time bucket and dimension key, then merge fine buckets into coarser ones, so long ranges read few rows and stay exact.›Graceful drainingsystems: On a termination signal, stop taking new work, finish in-flight work until a deadline, tell clients to reconnect elsewhere, then exit; clients resume from a token.›Cooperative schedulingconcurrency: Green threads/coroutines yielding at await points on a user-space schedulerFault injectiondistributed: Inject latency, errors, kills and partitions via proxies, netem or hooksRecord replaydistributed: Record every nondeterministic input (time, randomness, I/O, model and tool outputs, per-tick player input) and re-execute deterministically for debugging, replays and state rebuildLeasesdistributed: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.›Health checkingsystems: Decide per backend whether to route to it using probes and real-traffic errors, with consecutive-count hysteresis, capped ejection and a panic fallback.›Dynamic batchingml-training: Queue independent requests and dispatch them as one batch when it is full or the oldest has waited max_delay; bounded extra latency buys throughput.›Weight quantizationml-training: Store weights as 8/4-bit integers with per-group scales; decode is memory-bound, so 4-bit weights give about 3-4x tokens/s and fit 4x larger models.›Unicode text tokenization forsearchsearch-sort: Turn multilingual text into index terms: segment words (UAX 29 style, CJK bigrams), normalize, case-fold and accent-fold, identically at index and query time.›LU and Cholesky factorizationnumerics: Factor A = PLU (any square matrix, partial pivoting) or A = LLᵀ (SPD) once in O(n³), then solve each right-hand side in O(n²).›Gradient-boosted decision trees(GBDT)machine-learning: Add small regression trees one at a time, each fitted to the loss gradient; histogram splits over 256 bins make it the strongest tabular default.›Linear and logistic regressionmachine-learning: Fit a weighted sum of features to a number (least squares) or a probability (logistic); the baseline every model must beat.›Monte Carlo integrationnumerics: Estimate an expectation by averaging random samples; error falls as σ/√N in any dimension, and variance reduction (antithetic, control variates, importance sampling) shrinks σ.›+ 46 moreTransparent interceptionnetworking: Redirect traffic to a local proxy (iptables TPROXY/REDIRECT, eBPF) without app changesSyscall interceptionsecurity: Intercept syscalls via ptrace, seccomp user notify or a user-space kernelFencing tokensdistributed: A number that grows with every lock grant and is checked by the protected resource, so a paused ex-holder's late write is rejected.›MPMC bounded queueconcurrency: Vyukov's array queue for many producers and consumers: each slot carries a sequence number, so one CAS per operation claims a slot without locks.›Backpressuresystems: Bounded queues and credit-based flow so slow consumers throttle producersHierarchical timing wheelsystems: Timers bucketed by expiry into wheels of slots, coarser wheels for far deadlines: O(1) insert and cancel for millions of timeouts.›Low precision number formatsnumerics: Bf16, fp16, fp8 (E4M3, E5M2), MX block formats and int4: ranges, rounding, scalingText normalizationmachine-learning: Rewrite messy text into one canonical form for matchers and models, or into spoken words for speech output (and back), keeping a map to the original offsets.›Quantile sketchesdata-structures: Mergeable quantile sketches: t-digest, DDSketch, HDR histogramsRegularizationmachine-learning: Techniques that trade training fit for generalization: weight decay, dropout, early stopping, label smoothing and data mixing, with the exact update rules and defaults.›Loss functionsmachine-learning: The scalar a model minimizes: cross-entropy for classes and tokens, MSE or Huber for values, each with a stable form and simple gradient.›L-BFGSnumerics: Quasi-Newton minimizer that approximates the inverse Hessian from the last m gradient differences via the two-loop recursion; O(m·n) memory and work per step.›+ 18 moreEbpfhardware-os: Verified in-kernel programs on hooks (kprobes, XDP, tc) with mapsPtrace process controlhardware-os: Ptrace: attach, read and write registers and memory, single-step, intercept signalsSeccomp bpfsecurity: Syscall filtering with seccomp BPF programsFalse sharingconcurrency: Independent variables written by different cores on one cache line force the line to bounce between cores; pad or regroup hot data to fix it.›Memory orderingconcurrency: What Relaxed, Acquire, Release and SeqCst atomics actually guarantee, what they cost on x86 and ARM, and which one each pattern needs.›Slab allocatorsystems: Pre-carve pages into equal-size slots per size class with an in-place free list: O(1) alloc and free, no external fragmentation, cache-warm reuse.›+ 9 moreCPU caches and the memoryhierarchyhardware-os: Sizes, line widths and latencies of L1/L2/L3/DRAM on modern x86 and Apple M-series, and the data-layout rules that follow from them.›+ 1 more

How it is built

Architecture overview

An experience-management platform lets an organisation ask people questions at scale and turn the answers into decisions. It builds a questionnaire (customer survey, employee engagement pulse, product feedback widget), delivers it by email, link or in-app prompt to a defined audience, collects responses, and then reports: counts and averages by segment, trends over time, which topics drive the score, and which individual responses need action. Qualtrics, Medallia and Culture Amp are the known commercial products; LimeSurvey, Formbricks, KoboToolbox and ODK are open-source relatives.

Three ideas make it work. First, the questionnaire is data, not code: a declarative definition (questions, branching, validation, randomisation) is stored, versioned and interpreted by one engine on the server and on the client, so the same survey runs on web, mobile and in an email. Second, response data is append-only and analysed by scanning, so the store is columnar and pre-aggregated by time rather than a transactional row store. Third, the statistics are part of the product: a raw mean of whoever answered is misleading, so weighting, confidence intervals and significance tests sit between the data and the dashboard. A fourth, quieter concern is trust: respondents are promised anonymity, so reports must not allow a small group to be singled out.

Parts:

 admin ─► survey designer ─► definition (JSON, versioned) ─► validation + publish
                                          │
 audience (CRM, HRIS via SCIM) ─► sampler (hash / blocked) ─► invitations ─► durable queue ─► SMTP workers
                                                                       ▲ bounces, complaints, unsubscribes
 respondent ◄── link / email / in-app ── survey runtime (branching, quotas, randomisation)
      │ answers (append-only events, keyed by response id)
      ▼
 ingest ─► row store (responses, definitions)  ──► columnar facts ──► rollups by time / segment
                                                       │                      │
                         weights (raking) + intervals ◄┘                      ▼
                         text model (themes, sentiment) ──► dashboards, alerts, case routing
                         access control + minimum group size ──► what a viewer may see

How the parts fit together

Questionnaire engine: a definition and an interpreter

A survey definition lists pages, questions with types and constraints, and rules for what appears next. SurveyJS describes its library as rendering JSON-driven forms and surveys (source: SurveyJS repository), which is the common shape: a document that a runtime interprets. The document has a schema, and a JSON Schema validation step at publish time rejects malformed definitions before a respondent finds them. Branching and piping use small expressions ("Q3 > 7 and region = 'EU'"); a Pratt parsing or Recursive-descent parsing parser compiles them once at publish time, so the runtime evaluates a tree, not a string. Parsers are the one place where a survey author's input becomes code paths, so they are fuzzed (Coverage-guided fuzzing).

Assignment of respondents to variants (randomised question order, A/B wording, holdout groups) must be reproducible: the same person opening the link twice sees the same version. Deterministic bucketing does this by hashing the respondent id with a per-experiment salt. Experimental designs that need balance across strata (blocked or stratified randomisation) are randomization-blocked-stratified (no page yet), and conjoint studies need balanced sets of profiles (conjoint-choice-design (no page yet)). hash-based-sampling (no page yet) picks a fixed share of an audience so that all events of one person fall in or out together.

Distribution: email is the hard part

Most responses arrive from an email link, so deliverability is an engineering problem. Mail is sent by queue workers (email-delivery-smtp (no page yet) over the SMTP session of RFC 5321, smtp-protocol (no page yet)) pulling from a Durable job queue with Exponential backoff with jitter for temporary failures and a Token bucket per sending domain, because receiving servers throttle senders. The sender proves it is authorised with SPF, DKIM and DMARC sender authentication (RFC 7208, RFC 6376, RFC 7489), all of which are DNS lookups (DNS message format and client lookups).

Failures come back as delivery status notifications (RFC 3464) or feedback-loop complaints; bounce-processing (no page yet) classifies them (permanent, temporary, spam complaint) with pattern matching (Linear-time regular expression matching) over messages parsed by mime-parsing (no page yet), and suppresses the address after a hard bounce. List-unsubscribe handling follows RFC 8058. open-and-click-tracking (no page yet) records engagement through signed redirect links; an image fetch is not proof that a person read the message, so analyses should prefer clicks and completions as the engagement signal.

Identity, tenancy and access: one platform, many companies

Enterprise customers connect their own identity provider. saml (no page yet) (the OASIS SAML 2.0 specifications) delivers a signed XML assertion; the platform must parse it safely with a streaming parser (XML Streaming Parser), verify the signature with the provider's certificate (x509-certificates (no page yet), rsa-cryptosystem (no page yet)), and reject replayed assertions (Replay protection window). scim-provisioning (no page yet) (RFC 7643 and RFC 7644) keeps users and groups in sync: create, patch and deprovision calls from the customer's directory, validated against schemas (JSON Schema validation).

Data from different customers sits in shared tables, so isolation is enforced in the data layer: Row-level security (multi-tenant isolation in the data layer) appends a policy predicate to every query. Within a customer, Authorization models (RBAC, ABAC, ReBAC) decide who sees which results; organisational hierarchies ("a manager sees their team and below") are trees stored with a closure-table (no page yet). Traffic between services is protected by TLS 1.3 and, internally, Mutual TLS.

Response store and aggregation: scan, do not look up

Responses are written once and read in aggregate: "average score by region by week". Columnar storage stores each column contiguously so a query reads only the columns it touches. Low-cardinality columns (region, question id) compress with dictionary-encoding (no page yet), timestamps and counters with Delta encoding, and small integers with Bit packing and PFOR or Varint (LEB128). zone-maps (no page yet) keep per-block minimum and maximum so a time filter skips most blocks. Vectorized execution processes batches of values per operator call. Group-by uses hash-aggregation (no page yet).

Dashboards ask for the same aggregates repeatedly, so Rollup aggregation folds raw events into one mergeable partial per time bucket and segment, and Time partitioning keeps the buckets in time-ordered blocks so old data is cold and cheap. The row-oriented side (survey definitions, invitations, users) uses a B+ tree database behind connection-pooling (no page yet), because it is small, transactional and accessed by key. Two stores means a flow between them: the ingest path writes the response durably in the row store, and a batch or stream job appends it to the columnar store with an idempotent key, so a replay cannot double count.

Statistics: the part customers pay for

A sample of respondents is not the population. If younger employees answer less, a raw mean is biased. survey-weighting-raking (no page yet) adjusts weights so that weighted sample margins match known population margins (iterative proportional fitting); the R survey package documents post-stratification, calibration and raking as supported operations (source: CRAN survey package), and its variance estimation by Taylor series or replicate weights is what turns a weighted mean into a mean with an interval. Without weights, intervals for proportions use wilson-score-interval (no page yet), and for arbitrary statistics bootstrap-confidence-intervals (no page yet) resample the data. Hypothesis testing answers "did the score change after the programme?", with the caveat that dashboards that test dozens of segments will find false positives unless the number of comparisons is controlled.

Driver analysis ("what moves satisfaction?") fits Generalized linear models (GLMs) or Linear and logistic regression with the numeric tools that implement them: Newton root finding for the iteratively reweighted fit, LU and Cholesky factorization for the normal equations, and L-BFGS when regularised. Robust statistics (median, MAD) handle outliers in free-entry numeric answers. For small groups, a Bayesian hierarchical model fitted by markov-chain-monte-carlo (no page yet) borrows strength across groups; that costs minutes, not milliseconds, so it runs in a batch job. Monte Carlo integration is the simulation tool behind intervals and forecasts. Models that rank predictors use Gradient-boosted decision trees (GBDT), and shapley-values-treeshap (no page yet) turns a tree model's prediction into per-feature contributions that a business user can read.

Text analytics: themes and sentiment from comments

Open-text answers are often more valuable than scores. Text normalization cleans casing and spelling variants; Text classification assigns themes and sentiment, from keyword rules (Aho-Corasick multi-pattern matching) through linear models on vectors from a pretrained embedding model (embedding-inference (no page yet), run at serving time, no training of its own) to a fine-tuned Transformer block network, with Byte-pair encoding (BPE) tokenization handling many languages. Training uses AdamW and Reverse-mode automatic differentiation. Scores from the classifier are only useful as numbers if they are calibrated (Calibration), so "80 % negative" means what it says; this is the check to run before a model's output reaches an alert rule. rejection-sampling (no page yet) is the filter step when a generative model proposes summaries and a checker keeps the faithful ones.

Privacy: a minimum group size

Anonymous surveys promise that no one can be identified. Showing a team of two people's average reveals each person's answer to the other. k-anonymity (no page yet) sets a threshold: slices with fewer than k respondents are suppressed or merged, and two overlapping slices are also checked so that a difference of two reports cannot expose a small group. The check lives in the query layer next to row-level security, because dashboards and exports share it.

Durability and operation

Survey definitions and responses are customer data of long retention. They use Write-ahead log durability and ARIES recovery, Primary-backup replication for failover, and Point-in-time recovery with archives on Object storage. Workers are supervised (Process supervision) and drain before restart (Graceful draining). Invitation and ingest logic is tested under crashes and duplicates with Deterministic simulation testing.

hdr-histogram (no page yet) records the latency of the paths where respondents wait, loading the next question, saving an answer and the dashboard query, at fixed relative error. The 99th and 99.9th percentiles per tenant show whether a large survey launch is crowding out small customers, and they are the numbers the ingest and aggregation workers are scaled against.

Data and control flow

A survey goes out and results appear

  1. An administrator builds the survey. The definition is validated against the schema, its expressions are compiled, and a version is published; responses always reference the version they were collected under.
  2. The audience is loaded from the customer's directory (SCIM) or a file; a sampler picks recipients by hash, and the platform creates one invitation row per person with a signed, unguessable link.
  3. Invitations are queued. Workers send email at the allowed rate per domain, retrying temporary failures with jitter. Bounces and complaints update suppression lists.
  4. A respondent opens the link; the runtime serves the first page, evaluates branching as answers arrive, and saves partial progress. Each answer is an event with the response id, so saving twice is harmless.
  5. On completion the response is committed durably, and an idempotent job appends it to the columnar store and updates the time rollups.
  6. A dashboard request applies the viewer's permissions and row-level policies, reads rollups (and fresh data from the last partial bucket), applies weights and intervals, and suppresses slices below the minimum group size.
  7. A text job classifies new comments and routes strongly negative ones to a case queue, which sends an alert to the account owner.

An employee leaves the company

  1. The customer's directory sends a deprovisioning call (SCIM patch or delete).
  2. The platform deactivates the user, revokes sessions and removes their access to results; their historical responses stay, de-identified if the programme promised that.

Scaling

  1. Mail throughput and reputation. Sending is limited by what receivers accept; spread load across IPs and domains, authenticate, and ramp volume slowly.
  2. Response spikes. A launch email causes a burst of page loads and writes within minutes; serve the runtime statically, queue the writes, and keep the response write path to one durable insert.
  3. Aggregation cost. Move from scanning raw responses to rollups by time bucket and segment; add partitions and zone maps before adding hardware.
  4. Tenant skew. One large customer with millions of responses must not slow the others; partition by tenant, limit per-tenant query concurrency.
  5. Analytic models. Text models and Bayesian fits are batch work; precompute per survey and refresh on schedule.

Failure modes

FailureSymptomCauseFix
Invitations land in spamResponse rate drops suddenlyMissing or failing SPF, DKIM or DMARC, high complaint rateAuthenticate every sending domain, honour unsubscribes, suppress bounces
Bias in resultsDashboard score differs from realityLow response from some groups, no weightingWeight to known margins, show response rates and intervals
Double counted responsesTotals exceed audienceRetry of the ingest job without an idempotency keyKey by response id; replay-safe jobs
Small-group disclosureA manager can infer an individual's answerNo minimum group size, overlapping slicesSuppress below k, check overlapping reports, restrict exports
Cross-tenant data leakA user sees another customer's dataA query path without a tenant predicateRow-level security in the database, tests that query as each tenant
SSO assertion abuseAccount takeover through forged or replayed assertionsWeak XML signature handling, no replay checkStrict signature validation, reject replays, limit assertion lifetime
Expression bugRespondents see wrong or no questionsBranching error or edge case in expression evaluationTest definitions with generated paths, version definitions
Misleading significanceA "significant" change that vanishesMany segments tested, small samplesControl for multiple comparisons, show intervals not only p-values
Miscalibrated text modelSentiment alerts flood or stay silentModel trained on other domains, drift in vocabularyCalibrate on recent labelled data, monitor alert rates
Slow dashboardsSeconds per chart for large tenantsScans over raw responsesRollups, partitions, zone maps; precompute common cuts

Real implementations

namewhat it isrepolicencenotes
LimeSurveyOpen-source survey platformhttps://github.com/LimeSurvey/LimeSurveyGPL-2.0-or-later (LICENSE)https://www.limesurvey.org/ ; docs at https://docs.limesurvey.org/
FormbricksOpen-source experience-management platform ("Open Source Qualtrics Alternative")https://github.com/formbricks/formbricksAGPL-3.0 for most code; MIT for some packages; separate licence for ee directory (LICENSE)https://formbricks.com/docs
KoboToolbox (kpi)Server for KoboToolbox: forms, question library, sharing, exportshttps://github.com/kobotoolbox/kpiAGPL-3.0Forms, question library, sharing and exports via API
ODK CentralServer for Open Data Kit data collectionhttps://github.com/getodk/centralApache-2.0https://docs.getodk.org/central-intro/
SurveyJS Form LibraryJSON-driven survey and form renderer for web frameworkshttps://github.com/surveyjs/survey-libraryMITThe library, not the hosted service
R survey packageAnalysis of complex survey samples: weights, calibration, raking, variancehttps://cran.r-project.org/package=survey—See the CRAN page for the licence

Qualtrics, Medallia and Culture Amp are proprietary and not listed in the table.

Sources

  • SurveyJS Form Library repository (JSON-driven forms and surveys): https://github.com/surveyjs/survey-library
  • CRAN, survey: Analysis of Complex Survey Samples: https://cran.r-project.org/package=survey
  • RFC 5321, SMTP: https://www.rfc-editor.org/rfc/rfc5321 ; RFC 3464, delivery status notifications: https://www.rfc-editor.org/rfc/rfc3464 ; RFC 5322, message format: https://www.rfc-editor.org/rfc/rfc5322
  • RFC 7208 (SPF): https://www.rfc-editor.org/rfc/rfc7208 ; RFC 6376 (DKIM): https://www.rfc-editor.org/rfc/rfc6376 ; RFC 7489 (DMARC): https://www.rfc-editor.org/rfc/rfc7489 ; RFC 8058 (one-click unsubscribe): https://www.rfc-editor.org/rfc/rfc8058
  • RFC 7643 and RFC 7644 (SCIM): https://www.rfc-editor.org/rfc/rfc7644
  • OASIS, SAML 2.0 core: https://docs.oasis-open.org/security/saml/v2.0/saml-core-2.0-os.pdf

Similar systems

Share the most building blocks with this one.

customer-feedback 32 %customer-success 31 %forms-surveys 29 %crm 28 %influencer-marketing 27 %