← Back
Home › Systems › Payments and commerce

point-of-sale

Turns checkout events into a correct record of money and stock, even when the network, card reader, or power fails mid-sale.

Square

Open the point-of-sale pagepoint-of-sale Hdr histogramdata-structures: Log-linear bucketed histograms with bounded relative error for latency percentilesCRDTsdistributed: Data types whose replicas accept writes independently and always converge by merging, with no coordination: counters, sets, maps, registers, text.›Deterministic simulationtestingdistributed: Run the whole system in one thread under a seeded scheduler with simulated network, disk and clock, so every failure replays from its seed.›Incremental sync cursorsdistributed: The client keeps one opaque token; each pull returns only the changes after it plus the next token, so sync cost scales with what changed, not with the dataset.›Primary-backup replicationdistributed: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›Transactional outboxdistributed: Write the event as a row in the same local transaction as the state change; a relay publishes it later at-least-once, so state and events never diverge.›Dukpt key derivationfinance: DUKPT per-transaction key derivation for PIN pads and card readersEmv cryptogramsfinance: EMV ARQC/ARPC and tokenized-card cryptograms: session key derivation and MAC verificationIdempotency keysfinance: Client-chosen unique key per logical operation, stored with a request fingerprint and the final response, so retries replay the first result instead of acting twice.›Iso 8583 messagesfinance: ISO 8583 card messages: MTIs, bitmaps, data elements and their encodingsLargest remainder roundingfinance: Splitting an amount into parts that sum exactly (largest remainder / Hamilton) for allocations, splits and prorationsMoney representationfinance: Store money as integer minor units with an explicit currency exponent, compute in wider integers, and round once with a named rounding mode.›Payment state machinefinance: Persisted state machine for a payment lifecycle (authorize, capture, void, refund) that treats timeouts as unknown, retries with the same key and applies webhooks idempotently.›Process supervisionhardware-os: A parent process starts services, reaps their exits and restarts them under a delay and burst-limit policy, then stops them with SIGTERM before SIGKILL.›Coverage-guided fuzzingsecurity: Mutate inputs, run an instrumented target, and keep any input that reaches a new code edge; feedback turns blind guessing into incremental search.›Format-preserving encryptionsecurity: Encrypt a digit or character string into another string of the same length and alphabet with a keyed Feistel network, reversible with the key and a context tweak, without a lookup table.›Mutual TLSsecurity: TLS where the client also proves a certificate key, so every connection carries a verified, rotatable machine identity.›TLS 1.3security: The standard encrypted, authenticated transport; one round trip, only forward-secret AEAD suites, and little left to misconfigure.›B+ treestorage: Balanced high-fanout search tree with all values in linked leaves; the default on-disk index for point lookups and range scans.›Point-in-time recoverystorage: Periodic base backups plus a continuously archived change log let you rebuild the state as of any instant, then branch from there.›Write-ahead logstorage: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›Exponential backoff with jittersystems: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.›Graceful drainingsystems: On a termination signal, stop taking new work, finish in-flight work until a deadline, tell clients to reconnect elsewhere, then exit; clients resume from a token.›Fractional indexingdistributed: Give every list item a sortable string key and generate a new key strictly between its neighbours, so a move or insert writes one row.›Lamport clocksdistributed: One integer per process, max-plus-one on receive; gives a total order consistent with causality using 8 bytes per timestamp.›Vector clocksdistributed: One counter per node; comparing two vectors tells whether events are ordered or concurrent, which is exactly what conflict detection needs.›Cooperative schedulingconcurrency: Green threads/coroutines yielding at await points on a user-space schedulerFault injectiondistributed: Inject latency, errors, kills and partitions via proxies, netem or hooksRecord replaydistributed: Record every nondeterministic input (time, randomness, I/O, model and tool outputs, per-tick player input) and re-execute deterministically for debugging, replays and state rebuildMessage sequence numbersdistributed: A per-stream counter on every message lets the receiver detect loss, duplicates and reordering, ack cheaply and resume after reconnect with "send from N".›Cursor paginationnetworking: Keyset/opaque-cursor pagination for stable incremental reads from APIs and tablesHealth checkingsystems: Decide per backend whether to route to it using probes and real-traffic errors, with consecutive-count hysteresis, capped ejection and a panic fallback.›Event deduplication windowdistributed: Keep the ids of recently accepted events for a time window and drop repeats; memory stays bounded by rotating time buckets instead of per-entry timers.›Change data capturestorage: Decode the database's own commit log into row-change events in commit order, with a resume position, so any consumer can mirror the data exactly.›Leasesdistributed: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.›+ 17 moreTransparent interceptionnetworking: Redirect traffic to a local proxy (iptables TPROXY/REDIRECT, eBPF) without app changesSyscall interceptionsecurity: Intercept syscalls via ptrace, seccomp user notify or a user-space kernelConsumer offsetsdistributed: Each consumer group stores one integer per partition, the next position to read, so a restarted consumer resumes where it left off with at-least-once delivery.›Fencing tokensdistributed: A number that grows with every lock grant and is checked by the protected resource, so a paused ex-holder's late write is rejected.›+ 8 moreEbpfhardware-os: Verified in-kernel programs on hooks (kprobes, XDP, tc) with mapsPtrace process controlhardware-os: Ptrace: attach, read and write registers and memory, single-step, intercept signalsSeccomp bpfsecurity: Syscall filtering with seccomp BPF programsSegmented logdistributed: Split an append-only log into files named by their first offset, write only the newest, and find any record by binary search plus a sparse index.›+ 5 more+ 1 more

How it is built

Architecture overview

A point-of-sale (POS) system turns what happens at a counter (items scanned, a card tapped, a tip added, a refund given) into a correct record of money moved and stock sold, even when the network, the card reader or the power fails in the middle. It has two halves that fail differently. The front half is a terminal in a shop: a tablet or register, a card reader, a receipt printer, a local database, and a flaky uplink. The back half is a service that keeps the merchant's catalogue, orders, payments and reports, talks to the card network through an acquirer, and feeds accounting, inventory and loyalty. Everything in between is about one promise: a sale is recorded exactly once, a card is charged exactly once, and a crash at any point leaves a state that can be explained and repaired. Square is the best-documented example (its API docs describe the order, payment, terminal and webhook model), and the open-source ERP Odoo ships a POS module that follows the same shape at smaller scale.

Parts:

 shop                                                      cloud
┌───────────────────────────────────────┐   TLS 1.3 / mTLS   ┌──────────────────────────────────────┐
│ scanner/UI ─► order ─► payment engine │◄──────────────────►│ API ─► payment engine ─► acquirer    │
│                 │          │          │  sync cursor,       │          │   (ISO 8583 style messages)│
│            local DB (WAL)  ▼          │  idempotency keys   │       ledger/DB ─► outbox ─► CDC     │
│                 │     card reader     │                     │          │               │           │
│                 │  (EMV, PIN, DUKPT)  │                     │   replicas + PITR      event log     │
└───────────────────────────────────────┘                     │                     ┌──────┴──────┐  │
                                                              │         inventory  accounting  loyalty│
                                                              └──────────────────────────────────────┘

A POS system is therefore a small distributed system with an unreliable edge. The money path (card path, payment engine) is designed for correctness first; the read path (reports, dashboards) can lag.

How the parts fit together

Card path

The terminal never sees more of a card than it must. With a chip card, the card itself computes a cryptogram over the transaction data (EMV's ARQC for an online authorisation request; the issuer's answer can carry its own cryptogram, the ARPC), so the issuer can tell a genuine card from a copied magnetic stripe. The primitives behind that are EMV session-key derivation and MAC checks (emv-cryptograms, no page yet). The PIN, if entered, is encrypted inside the reader's tamper-resistant hardware. Readers typically derive a fresh key per transaction from a base key with DUKPT (dukpt-key-derivation, no page yet; ANSI X9.24), so stealing one terminal's current key does not decrypt earlier transactions.

Between the terminal, the acquirer and the card network, authorisation messages follow the ISO 8583 family (iso-8583-messages, no page yet): a message type indicator, a bitmap that says which data elements are present, then the elements themselves. Requests, advices and reversals are different message types; the reversal matters for the payment engine below, because "I asked and heard nothing" must be answerable with "undo it".

Two rules keep the blast radius small. First, the card number is stored, if at all, encrypted or replaced by a token; Format-preserving encryption lets a 16-digit number become another 16-digit string so that old fields, validators and reports keep working. Second, anything that compares secrets (MACs, PIN blocks) must be constant-time (constant-time-code, no page yet) and every key and nonce must come from the operating system's secure random source (csprng, no page yet). The PCI Security Standards Council publishes the rules that decide how much of this a merchant system has to implement itself (see Sources).

Payment engine

The Payment state machine is the heart of the money path. A payment is a persisted object with states such as created, authorized, captured, voided, refunded and failed. The central design decision is how to treat a timeout: it is not a failure, it is "unknown". The engine records the unknown state, then asks the acquirer for the outcome or sends a reversal, and only then moves on. Nested states keep this readable: a Hierarchical state machine lets "any in-flight state" share one handler for "connection lost" instead of repeating it in every state.

Every call that changes money carries an Idempotency key: a client-chosen unique string for the logical operation, stored with the request fingerprint and the final response, so a retry returns the first result rather than charging again. Square documents the same pattern for its APIs (an idempotency key on create calls) and Stripe's engineering blog explains why (see Sources). Retries use Exponential backoff with jitter, so a hundred terminals that lost the same uplink do not retry in lockstep and knock over the recovering service.

Contracts with its neighbours:

  • To the card path: one authorisation request per attempt; a reversal for every authorisation whose outcome is unknown; the network's response code is stored verbatim next to the engine's own state.
  • To the order: a payment belongs to one order; the order cannot be closed until captured amounts equal the total, tips included. Tip adjustment is a capture for a different amount than the authorisation, within the network's rules.
  • To the event backbone: every state change is also an event (below), written in the same transaction.

Money and totals

Money representation fixes the unit: integer minor units (cents) plus an explicit currency, never floats. Taxes, discounts and service charges are computed in wider integers and rounded once, with a named rounding mode, at the point the rules say. When an amount has to be divided (a bill split four ways, a tip pooled among staff, a discount spread over line items), the parts must sum to the whole exactly; the largest-remainder method (largest-remainder-rounding, no page yet) hands the leftover cents to the parts with the biggest fractional remainders. The reason to be strict is auditing: a one-cent drift per sale across millions of sales is a reconciliation break that nobody can explain.

Local store and offline sync

The terminal keeps its own database so a sale can finish with no network. That database is crash-safe for the reasons in Write-ahead log (log, checksum, fsync, then update pages), usually built on a B+ tree for catalogue lookups, with CRC32C checksums on blocks and log records to detect torn writes.

The sync protocol is the contract between terminal and server. With Incremental sync cursors the terminal holds one opaque token; each pull returns the changes after it plus the next token, so sync cost scales with what changed, not with the size of the catalogue. Message sequence numbers on the push side let the server detect gaps and duplicates in the terminal's outgoing sales; cursor-pagination (no page yet) bounds each page. Sales are append-only facts and are rarely in conflict. Configuration data that two people can edit while offline (a menu item's price at the shop and at head office) is where conflict handling begins: CRDTs with Vector clocks or Lamport clocks give merge rules that converge without coordination, and Fractional indexing keeps a re-ordered menu to one changed row. Many systems skip CRDTs and choose a plain rule ("head office wins, last write per field") with Optimistic concurrency control: read a version, write only if it is unchanged.

Event backbone

Inventory, accounting, loyalty and receipts must learn about a sale, and none of them may be allowed to block it. The Transactional outbox writes the event as a row in the same local transaction as the state change; a relay publishes it afterwards at-least-once. Either the sale and its event both exist or neither does. The relay can be a poller or Change data capture, which reads the database's own commit log in commit order with a resume position. Consumers receive duplicates by design, so they apply Event deduplication (a time-windowed set of event ids) and track progress with Consumer offsets. Where the business wants history rather than state, Event sourcing keeps the order or payment as an append-only stream of events and derives current state by folding over it; the Segmented log is the storage shape of such a stream. The loyalty system is one such consumer.

Secure transport

Every link from terminal to server is TLS 1.3: one round trip to set up, forward-secret key exchange (ECDH key exchange, keys expanded with HKDF key derivation) and authenticated encryption (AEAD), with little to misconfigure. Server certificates follow x509-certificates (no page yet). Terminals are not anonymous browsers, so Mutual TLS gives each one a rotatable client certificate, and the server can revoke a stolen terminal by revoking its identity. SHA-2 and BLAKE3 cover fingerprints, request signing and webhook verification (Square documents how to verify webhook notifications; see Sources).

Durability and recovery

On the server, Primary-backup replication orders all writes on one node and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash can lose. For payments the rule is strict, because an acknowledged capture that disappears is a financial incident. Leases decide which node is primary, Health checking decides when to fail over, and Fencing tokens make sure a deposed primary that wakes up late cannot write. Point-in-time recovery (base backups plus continuous log archive, with Object storage as the target and ARIES recovery as the crash-recovery model beneath it) answers the human failure: rebuild the state as of ten minutes before the bad deploy. The Buffer pool is where the database decides which dirty pages reach disk and when, which is why its write ordering is part of the durability story.

Verification

Deterministic simulation testing runs the whole system under a seeded scheduler with simulated network, disk and clock, so any failing run replays from its seed (PRNG; record-replay and fault-injection have no page yet). Coverage-guided fuzzing attacks the parsers that receive outside bytes, such as the ISO 8583 decoder, using Code coverage instrumentation; delta-debugging (no page yet) minimises failing inputs.

Operations

Process supervision restarts crashed services under a delay and burst limit. Graceful draining lets a service being replaced finish in-flight work and tell clients to reconnect; terminals resume from their cursor. hdr-histogram (no page yet) records authorisation latency percentiles with bounded error.

Data and control flow

One card sale, online:

  1. The cashier builds an order; totals and tax are computed in integer cents and written to the local database.
  2. The terminal creates a payment with a fresh idempotency key and sends it over mutual TLS.
  3. The server records the payment as created, then asks the acquirer to authorise (ISO 8583 style request).
  4. The reader runs the chip dialogue and PIN entry; the cryptogram travels with the request.
  5. The response moves the payment to authorized or failed. If the reply does not arrive, the state is unknown and a reversal or status query follows with the same key.
  6. Capture (immediately or at close of batch) moves the payment to captured and closes the order. The state change and its event commit in one transaction.
  7. The relay publishes the event; inventory, accounting, loyalty and the receipt service each apply it once.

One sale, offline:

  1. The order is stored locally; the card is accepted within the offline limits, or the sale is cash.
  2. The terminal queues the payment with its sequence number.
  3. When the uplink returns, the terminal pushes the queue in order, then pulls changes using its cursor.
  4. The server replays the payments through the same state machine; declines after the fact become exceptions for the merchant to handle.

Scaling

Limits, in the order they bite, and the lever for each:

  1. Per-terminal latency (the customer is waiting) is bound by the acquirer round trip, not by your code. Levers: keep connections open, avoid extra hops, measure percentiles, keep authorisation off the critical path of anything that can wait.
  2. Peak hour per merchant (lunch rush, Black Friday): the payment engine is a write path with strict ordering per payment. Lever: partition by merchant or payment id; the state machine needs no ordering across payments.
  3. Fan-out of events grows with the number of consumers. Levers: the commit log with consumer offsets lets each consumer proceed at its own pace; add consumers without touching the write path.
  4. Sync storms when a region comes back online and thousands of terminals pull at once. Levers: cursors keep each pull small, jittered backoff spreads the retries, draining protects a restarting node. No published figure is used here; measure your own peak. The shape is stable, though: one location is a low-rate write path, a platform of many merchants is a high-rate one, so partitioning by merchant is the first scaling step and rarely the last.

Failure modes

FailureSymptomCauseFix
Double chargecustomer charged twice, one orderretry without an idempotency key, or key reused across different amountskey per logical operation, store the request fingerprint, reject a key reused with a different body
Lost responsepayment shows failed but the money moveda timeout treated as a declinepersist "unknown", query status or reverse before any new attempt
Sale without eventorder paid, inventory not updatedevent published outside the transaction, crash between the twotransactional outbox; relay at-least-once
Duplicate eventstock decremented twiceat-least-once delivery, consumer not idempotentevent ids plus a deduplication window
Cent driftdaily totals differ from the bank by small amountsfloats, or rounding per line instead of onceinteger minor units, rounding mode named in one place, largest-remainder for splits
Offline pile-upmany declines after reconnectoffline limits too high, stale deny listsconfigurable limits, report offline exposure, forward in sequence order
Split braintwo primaries accept paymentsfailover without fencingleases plus fencing tokens checked by storage
Stolen terminalfraudulent requests with a valid identitylong-lived client certificateshort-lived mTLS certificates, revocation, per-terminal limits

Real implementations

namewhat it isrepolicencenotes
Squarehosted POS with documented APIs for orders, payments, terminal and webhooksSquare developer docsproprietary servicethe reference for the API shape; Terminal API, Sandbox
Odoo Point of SalePOS module of an open-source ERP, runs a web UI with offline modeodoo/odoosee repoone codebase for POS, stock and accounting
Floreant POSopen-source restaurant POSfloreantpos/floreantpossee repodesktop application, local database
Apache OFBizopen-source business framework with POS, orders and accountingapache/ofbiz-frameworksee repoenterprise framework rather than a counter product
Debeziumchange data capture for the outbox relaydebezium/debeziumsee reporeads the database commit log
Apache Kafkapartitioned commit log for the event backboneapache/kafkasee repoconsumer offsets and retention as described above

The alternative: let a payment provider own the card path and payment state machine and keep only orders and sync. You integrate through its API and webhooks instead of ISO 8583 and EMV, and most PCI scope moves to the provider. The price is fees, their outage becoming yours, and less control of offline behaviour.

Sources

  • Square, Idempotency (common API patterns): docs; Webhooks: docs; Terminal API: docs.
  • Stripe, Designing robust and predictable APIs with idempotency: blog.
  • EMVCo, EMV contact chip technology: emvco.com.
  • ISO 8583 overview: Wikipedia (message type indicator, bitmap, data elements).
  • PCI Security Standards Council: pcisecuritystandards.org.
  • The storage, security and distributed-systems blocks above: see their own pages for their sources.

Similar systems

Share the most building blocks with this one.

card-payments 48 %payment-gateway 48 %offline-first-mobile 44 %billing 42 %card-issuing 40 %