mainframe-tp-monitor

How it is built

Architecture overview

A transaction processing (TP) monitor is the runtime that sits between thousands of terminals or network clients and a set of data stores, and turns each short request into an all-or-nothing unit of work. IBM CICS and IMS TM on the mainframe, and Oracle Tuxedo on Unix, are the classic examples. The application programmer writes a small program per transaction; the monitor owns everything around it: accepting the request, queuing and routing it to a free worker, giving it a task with its own storage, calling the databases and files it needs, and committing or backing out every change together. IMS TM, for example, stores an incoming transaction on a message queue and its scheduler then starts the application in a message processing region (source: Wikipedia, IBM Information Management System).

The one idea that makes it work is separating the transaction from the process. Operating-system processes and threads are expensive, so the monitor keeps a small pool of long-lived server processes and multiplexes many short transactions over them, each with its own storage and a unit-of-work identity that all data stores honour. Atomicity across several data stores comes from a transaction manager running two-phase commit over a standard interface (X/Open XA). Availability comes from splitting the monitor into several cooperating regions or servers, so that a failure of one application server does not take down the terminal network, and from a shared lock and cache facility when one machine is not enough.

Parts:

• Front end (terminal or listener) — owns connections, authentication, and routes each request to an application server — Mutual TLS, Authorization models, connection-pooling (no page yet), backpressure (no page yet)
• Dispatcher and task runtime — queues requests, gives each a task, keeps the server processes alive — Context switching, Process supervision, Graceful draining
• Service queues — named services, in-memory and persistent queues, retries — Durable job queue
• Transaction manager — begins, prepares, commits or rolls back global transactions across data stores — Two-phase commit
• Resource managers — the databases and files the application touches — Buffer pool, B+ tree, Two-phase locking, Deadlock detection, Index-free adjacency
• Recovery and replication — restart after a crash, standby for a lost machine, restore to a point in time — ARIES recovery, Point-in-time recovery, Primary-backup replication
• Cluster data sharing — several machines work on the same data at once — shared-data-cluster-lock-cache (no page yet), Cache invalidation, Fencing tokens
• Batch window — end-of-day and settlement jobs next to online work — batch-checkpoint-restart (no page yet)
• Operations and verification — latency histograms, usage accounting, checksums, and tests of the recovery paths — hdr-histogram (no page yet), resource-usage-accounting (no page yet), CRC-32C checksums, Deterministic simulation testing, Coverage-guided fuzzing, fault-injection (no page yet)

 terminals / network clients
        │  authenticate, route
        ▼
 ┌───────────────┐   queue      ┌────────────────────────────────────────┐
 │ front end     │────────────► │ application server(s) / regions        │
 │ (TOR, listener)│  by service  │  task per request, own storage        │
 └───────────────┘              │  program calls: read, update, enqueue  │
        ▲ reply                 └───────┬───────────────────┬────────────┘
        │                          begin│ commit / abort    │ data calls
        │                               ▼                   ▼
        │                      ┌──────────────────┐   ┌─────────────────┐
        └──────────────────────│ transaction mgr  │◄─►│ resource mgrs   │
                               │ 2PC + decision   │XA │ (DB, files, MQ) │
                               │ log              │   │ locks + WAL     │
                               └────────┬─────────┘   └────────┬────────┘
                                        │ recovery log         │ shared locks/cache
                                        ▼                      ▼
                                  standby / restart      cluster coupling

How the parts fit together

Front end: terminal-owning region and listener

The front end holds the network side: terminal sessions, TCP connections, TLS. In CICS this is the terminal-owning region (TOR), which routes transactions to several application-owning regions (AORs), and a typical installation has several TORs for each service (source: Wikipedia, CICS). Keeping the connection state out of the application servers is the design point: an application server can crash or be upgraded while the sessions stay up. Identity arrives here, so Mutual TLS authenticates machine-to-machine links and Authorization models decide which user may start which transaction code. connection-pooling (no page yet) and backpressure (no page yet) are the two controls on the way in: bound the number of open sessions per client and refuse early rather than queue without limit.

Dispatcher and task runtime

The runtime gives each request a task with private working storage and runs many tasks on few operating-system threads. Within a CICS region transactions are cooperatively multitasked: they are expected to yield rather than block, and CICS services do this automatically (source: Wikipedia, CICS). The reason is cost: a Context switching between processes saves registers and address space and disturbs caches, while a yield at a known point in a user-space scheduler (cooperative-scheduling, no page yet) costs little. The price is the failure case in Failure modes: one task that does not yield blocks the whole region. A second price: because of its quasi-reentrant programming model, a CICS address space could use only about 1.5 processors at the time of the early-1990s Sysplex work, so capacity came from more regions, not more threads (same source). Process supervision restarts a failed server under a burst-limit policy, and Graceful draining lets a server finish in-flight tasks before a stop.

Tuxedo does the same job with processes on Unix. Its Bulletin Board is a shared-memory segment holding the configuration and state of a domain (servers, services, transactions, clients), a bulletin board liaison (BBL) process on each machine coordinates updates to it, and a Bridge process passes requests between machines; the BBL and Bridge monitor each other and restart the other on failure (source: Wikipedia, Tuxedo). Requests are queued to servers through memory-based inter-process communication, and the client does not know where or how the server runs (same source).

Service queues and messaging

Between the front end and the application sits a queue per service. IMS TM puts the incoming transaction on a queue, in memory or in a dataset, and the scheduler starts the program in a message processing region (source: Wikipedia, IBM Information Management System). Tuxedo adds a /Q component with transient (in-memory) and persistent queues and an event broker for publish and subscribe (source: Wikipedia, Tuxedo). A persistent queue is a Durable job queue: messages survive a restart, a worker takes a lease and acknowledges on completion, and a failed message is retried with Exponential backoff with jitter and parked in a Dead-letter queue after a limit. Message sequence numbers let a receiver detect loss, duplicates and reordering on a link; Idempotency keys make a retried request return the stored answer instead of repeating the update. An in-process bus (publish-subscribe-event-bus, no page yet) carries events inside a region.

Transaction manager and the XA contract

The contract between the monitor and the data stores is the X/Open DTP model and its XA interface, published by The Open Group. It names the calls the transaction manager uses on a resource manager: xa_start and xa_end to bracket the work of a branch, xa_prepare, xa_commit and xa_rollback for the two phases, xa_recover to list prepared branches after a crash, and xa_forget for heuristic outcomes (source: The Open Group, Distributed Transaction Processing: The XA Specification). The spec also describes a read-only optimisation, a one-phase commit when only one resource manager takes part, and heuristic decisions by a resource manager that has prepared, which it must report to the transaction manager and not forget (same source). XA is based on an interface used in Tuxedo in the 1980s (source: Wikipedia, X/Open XA), and CICS supports XA two-phase commit with Db2, IMS/DB, MQ and VSAM with record-level sharing, as well as SNA LU 6.2 for transactions across address spaces (source: Wikipedia, CICS).

Two-phase commit is the algorithm. Every participant durably records its yes vote before replying, and the coordinator logs its decision before telling anyone; both are forced writes to a Write-ahead log, and the force is the fsync and direct I/O call whose real behaviour per platform decides whether a "committed" transaction survives a power cut. The weak point is known: 2PC blocks. If the transaction manager is down after participants have prepared, they hold their locks until it returns, and if its decision log is lost an operator must resolve branches by hand (source: Wikipedia, X/Open XA).

Resource managers: locks, buffer pool and index

Each resource manager is a database engine or file system in its own right. It reads pages through a Buffer pool that owns eviction (LRU and CLOCK), finds rows through a B+ tree, and isolates transactions with Two-phase locking plus Deadlock detection. The monitor does not look inside; it only needs the XA calls and a lock timeout policy so that a stuck branch does not hold up the others.

Not every resource manager is relational. A hierarchical database of the IMS kind keeps each record's child and sibling addresses inside the record, and Index-free adjacency is what makes a transaction's typical call, "get the next dependent of this parent", a direct pointer read through the Buffer pool instead of an index probe. The transaction manager sees this store through the same prepare/commit calls as any other, and a B+ tree index is added only for the lookups the pointers do not serve.

Recovery and replication

After a crash, ARIES recovery repeats the logged history in the buffer pool and rolls back the transactions that had not committed, which is what the resource manager does on its own restart; the transaction manager then asks it for in-doubt branches (xa_recover) and resolves them from its decision log. CICS regions are restarted in a "warm" mode that uses the checkpoint written at shutdown, or a "cold" mode that reloads definitions and discards the log (source: Wikipedia, CICS). For a lost machine, a standby follows through Primary-backup replication; a Lease decides which node is allowed to be primary, and Health checking decides when to fail over. Point-in-time recovery covers the case replication cannot: a bad batch run or a deleted table is copied to the standby as faithfully as good data.

Cluster data sharing

Splitting one workload over several machines creates the sharing problem: two machines must not update the same record or cached page without knowing about each other. In IBM's Parallel Sysplex the answer is a coupling facility holding lock, list and cache structures that serialise access, pass messages and keep buffers consistent between logical partitions (source: Wikipedia, IBM Parallel Sysplex). The general technique is shared-data-cluster-lock-cache (no page yet). On a commodity cluster the equivalent pieces are Cache invalidation, so a write on one node removes copies on the others, and Fencing tokens, so a node that lost its lock during a pause cannot write afterwards.

Operations and verification

Latency percentiles go into a bounded-error histogram (hdr-histogram, no page yet), CPU and I/O are charged per transaction type (resource-usage-accounting, no page yet), and log records carry a checksum (CRC-32C checksums). Batch jobs (batch-checkpoint-restart, no page yet; cron-scheduling, no page yet) must agree with online work on commit frequency, or a long batch transaction blocks the terminals. The recovery paths are the hardest to test by hand; Deterministic simulation testing replays crashes, delays and disk faults from a seed, and Coverage-guided fuzzing attacks the message and buffer parsers at the front end.

Data and control flow

One transaction, from terminal to reply:

1. The user sends a transaction code and input. The front end authenticates the session, maps the code to a service and places the request on that service's queue (order of microseconds in memory).
2. The dispatcher assigns the request to a free server or region; the task gets its own storage and a unit-of-work identity.
3. The program starts a global transaction. For each data store it touches, the transaction manager calls xa_start; the data store takes locks as it reads and writes, and appends change records to its log.
4. The program ends its part and asks to commit. The transaction manager calls xa_end, then xa_prepare on every branch. Each branch forces its log and answers yes or no (the dominant cost: one log force per branch).
5. If all say yes, the transaction manager forces its decision record, then calls xa_commit on each. Locks are released and the reply is sent to the front end, which writes it to the terminal.
6. The transaction manager forgets the transaction after all acknowledgements; read-only branches skip phase two.

A crash between the phases:

1. A data store restarts and uses its log to redo committed work and undo unfinished work; branches that had prepared stay in doubt and keep their locks.
2. The transaction manager restarts, reads its decision log, and calls xa_recover on each resource manager to list in-doubt branches.
3. For each branch it resends the logged decision (commit if a commit record exists, else rollback). If its log is gone, an operator resolves the branch by hand, which is the heuristic path in the XA specification.

Scaling

In the order you hit them:

1. Log forces per transaction. Each commit waits for at least one durable log write per participant plus the coordinator. Levers: group commit (many transactions share one force), a battery-backed or NVMe log device, fewer participants per transaction, one-phase commit when only one resource manager is involved, the read-only optimisation.
2. Lock contention on hot rows. A counter or an account record that every transaction updates serialises them. Levers: shorter transactions, commit earlier, split the hot record, move to optimistic schemes where collisions are rare.
3. Single dispatcher or region capacity. One CICS address space could not use more than about 1.5 processors in the early 1990s (source: Wikipedia, CICS). Lever: more regions behind the same front end, with transactions routed by the front end to several application regions, which the mainframe world calls dynamic routing.
4. Shared data across machines. Lock and cache traffic between nodes. Lever: the hardware-assisted coupling facility, or on commodity hardware, routing by key so each record has one home node and cross-node locks are rare.
5. Front-end connections. Many mostly idle sessions. Lever: terminal-owning regions or listener processes with their own pools, separate from application capacity.
6. Recovery time. More log means longer restart. Lever: frequent checkpoints, a warm standby that applies the log continuously.

Failure modes

Real implementations

The alternative: most new systems avoid distributed transactions. A single database transaction covers one service's data; across services, designs use a durable queue with idempotent consumers or a saga with compensating steps. A TP monitor with XA earns its place when several heterogeneous stores must commit together and the transaction rate is high, which is the profile of banking cores, airline reservation and telecom billing.

Sources

• The Open Group, Distributed Transaction Processing: The XA Specification. PDF
• Wikipedia: CICS, Tuxedo (software), IBM Information Management System, IBM Parallel Sysplex, X/Open XA
• Narayana: site; Jakarta Transactions: specification page
• Enduro/X: repository README, Mavimax
• PostgreSQL documentation, PREPARE TRANSACTION