energy-retail-billing

How it is built

Architecture overview

An energy retailer sells electricity or gas it does not deliver. The wires and pipes belong to a network operator, the meters belong to somebody else, and the wholesale market settles the retailer's volume in its own cycle. The retail system (a customer information and billing system, CIS) turns three streams into one bill per customer: who the customer is and which contract applies (accounts, tariffs, meter points), how much they used (meter reads and interval data), and what the market and the network charge (levies, network fees, taxes). It then collects the money and keeps books that an auditor can follow.

The idea that makes it work is that a bill is a pure function of recorded facts. Meter reads, tariff versions, price components and customer-contract events are stored as immutable, time-stamped records; billing reads them and computes. A correction, such as a late meter read or a retroactive tariff change, does not edit a bill: it adds a fact, re-runs the function for the affected period, and posts the difference as a credit or debit. This is why event sourcing, a double-entry ledger and idempotent batch runs show up in every serious implementation.

Parts:

• Meter data intake — receive reads from smart meters, meter operators and market messages without loss or duplication — Message sequence numbers, Consumer offsets, Segmented log, CRC32C checksums, Varint (LEB128), Delta encoding, x12-edi-messaging (no page yet), iso-20022-messages (no page yet)
• Meter data store — years of interval readings per meter, scanned by billing and analysis — Columnar storage, Bit packing and PFOR, dictionary-encoding (no page yet), zone-maps (no page yet), Vectorized execution, LSM-tree, SIMD programming, Slab allocator
• Validation and estimation — find bad reads, fill gaps with estimates — Time-series anomaly detection, STL decomposition (seasonal-trend with loess), Robust statistics, Isolation forest, Exponential smoothing (ETS, Holt-Winters), spline-interpolation (no page yet)
• Tariff and rating — price consumption under the contract in force at the time — tariff-rating (no page yet), Interval tree, Money representation, largest-remainder-rounding (no page yet), calendar-date-arithmetic (no page yet), proration-calculation (no page yet)
• Billing run — generate thousands to millions of bills on a schedule and survive crashes — Durable workflow execution, Durable job queue, Leases, Fencing tokens, Dead-letter queue, Hierarchical timing wheel, Windowed stream aggregation, watermarks (no page yet), hash-aggregation (no page yet), d-ary heap, record-replay (no page yet)
• Accounts and ledger — keep every charge, payment and adjustment balanced — Double-entry ledger, gapless-sequence-numbers (no page yet), revenue-recognition-schedules (no page yet), Optimistic concurrency control, Write-ahead log, fsync, O_DSYNC and O_DIRECT
• Customer lifecycle and collections — move in, switch supplier, pay, dunning — Event sourcing, Hierarchical state machine, Payment state machine, Idempotency keys, Exponential backoff with jitter
• Integrity and test data — fingerprint documents, generate synthetic customers — Cryptographic hashes (SHA-2, BLAKE3), PRNGs (PCG, xoshiro)

 meters / meter operators ─► intake (dedupe, checksum) ─► raw read log ─► validate + estimate ─► interval store (columnar)
 market & network messages ─► intake                                                                    │
                                                                                                        ▼
 contracts, tariffs (versioned) ───────────────────────────────────────────────►  rating (usage × price, by interval)
 customer events (move-in, switch, price change) ──► event log ──► account state          │
                                                                                          ▼
 billing scheduler ─► durable job queue ─► bill worker ─► invoice + ledger postings ─► ledger (double entry)
                              ▲                                   │                          │
                              └── retries, dead letters           └─► documents, e-mail      └─► payments, dunning, finance

How the parts fit together

Meter data intake

Smart meters record consumption at fixed intervals (in the field commonly 15 or 30 minutes, which is 96 or 48 values per meter per day) and send them to a central head-end system; a meter data management system then imports, validates and cleanses them before billing (Wikipedia, Meter data management). Readers speak meter protocols such as DLMS/COSEM; open libraries like Gurux's exist for reading meters directly. Market data arrives as batch files or messages in sector formats, often EDI-style (x12-edi-messaging, no page yet) or XML (iso-20022-messages, no page yet, for payments).

Intake has three duties. It must make duplicates harmless: the same interval can arrive from the meter, again from a re-send and again from a corrected file, so each reading carries a key of meter point, channel and interval start, and a Message sequence number per source shows gaps. It must keep every raw record, because disputes are decided on what was received, not on what was cleaned. And it must be cheap to replay: raw reads go to an append-only Segmented log with per-record CRC32C checksums, and downstream stages keep Consumer offsets, so a stage that crashes or is rewritten resumes or replays from a stored position. Compact encodings (Varint (LEB128) for small deltas) pay off because the volume is large and the values change slowly.

Meter data store

The access pattern decides the layout. Billing asks, for one meter point and one period, for all intervals; analysis asks for one hour across all meters. Neither is an OLTP point read. Columnar storage keeps each field contiguous; timestamps become small deltas (Delta encoding), values fit in few bits (Bit packing and PFOR), repeated codes such as quality flags go into dictionaries (dictionary-encoding, no page yet), and per-block min/max (zone-maps, no page yet) skip blocks that cannot match. Vectorized execution scans batches and, where it pays, SIMD programming does the arithmetic. New reads arrive as small writes in time order, which an LSM-tree turns into sorted immutable runs; a Slab allocator keeps per-read objects off the general heap in the ingest path.

The contract with billing is a read API: "give me quality-flagged intervals for meter point M from t1 to t2 as of version V". The as of matters, because corrected reads supersede earlier ones without deleting them.

Validation and estimation

Raw reads are unreliable: meters reset, clocks drift, a communication outage leaves gaps. The validation, estimation and editing stage (VEE in the field) does three things in order. It flags implausible values, using a seasonal baseline per meter: STL decomposition separates daily and weekly shape, Robust statistics (median and MAD) scale the residual so one spike does not distort the limit, and an Isolation forest catches multivariate oddities across channels (this is the pattern of Time-series anomaly detection). It fills gaps with estimates drawn from the meter's own history: Exponential smoothing for short gaps, similar-day profiles or spline-interpolation (no page yet) for longer ones. And it marks every value as measured, estimated or edited, so the bill can say "estimated read" and a later real read can trigger true-up.

Tariff and rating

A tariff is a price plan valid over a date range: a standing charge per day, one or more unit prices per kWh, time-of-use bands (cheaper at night; Wikipedia, Time-of-use pricing), tiers, taxes, and pass-through network charges. Rating applies the plan to each interval. Because plans change, and because a bill period can span a price change, the rater needs the plan in force at the interval's time. An Interval tree answers "which tariff versions overlap this period" in O(log n + k). A move or plan change mid-period needs proration (proration-calculation, no page yet) of fixed charges, and dates need care with time zones and daylight saving (calendar-date-arithmetic, no page yet): a day has 23 or 25 hours twice a year, and an interval store keyed in local time breaks twice a year if this is ignored.

Money is computed with Money representation: integer minor units or fixed-point with a stated rounding rule. Energy prices are often defined with more decimals than the currency's minor unit (fractions of a cent per kWh), so the rule must say where rounding happens (per interval, per line, per invoice). Splitting an amount over several lines so the parts sum exactly uses largest-remainder-rounding (no page yet).

Billing run

A billing run touches every customer in a cycle, so it is a batch of many independent small jobs. The scheduler enqueues one job per account into a Durable job queue: workers take a job under a Lease that expires if the worker dies, retry with Exponential backoff with jitter, and park poison jobs in a Dead-letter queue for a human. A paused or restarted worker holding a stale lease is stopped by Fencing tokens checked at the ledger. A Hierarchical timing wheel holds the many future timers (next read due, payment due dates, retry times).

Each bill job is a multi-step workflow: fetch reads, rate them, add charges, post to the ledger, render the document, send it. Durable workflow execution logs the result of each step and replays after a crash, so a job that died after posting but before sending does not post again. Aggregates that span accounts (revenue per tariff, volume per network area) use Windowed stream aggregation with event-time windows and watermarks (no page yet); intermediate grouping uses hash-aggregation (no page yet). Replaying a run for audit is easiest when the inputs are fixed: record-replay (no page yet).

Accounts and ledger

Each bill, payment, adjustment and write-off is a set of postings in a Double-entry ledger: debits and credits sum to zero per currency, so a defect shows up as an imbalance instead of a silent error. Invoice numbers are usually legally required to be gap-free (gapless-sequence-numbers, no page yet), which means numbers are assigned inside the transaction that commits the invoice, not up front. Concurrent updates to one account (a payment arrives during a bill run) are resolved by Optimistic concurrency control: read a version, write if it is unchanged, else retry. Durability comes from the Write-ahead log, and the cost of fsync, O_DSYNC and O_DIRECT is the floor for commit latency. Deferred revenue and unbilled energy are recognised by schedules (revenue-recognition-schedules, no page yet).

Customer lifecycle and collections

A customer's relationship to the retailer is a long-lived state machine: lead, quote, switch requested, pending start date, active, moved out, final bill, closed. Hierarchical state machine fits because many events behave the same in all "active" substates (a meter read, a price change), and only a few differ. The state is stored as an Event sourcing stream of facts such as "move-in on 3 March", "plan changed", "read corrected", from which the account's current state, and its state on any past date, is a fold. This is what makes retroactive corrections tractable: change the facts, re-fold, re-rate, post the difference.

Payments follow the Payment state machine (authorise, capture, fail, refund, chargeback) with Idempotency keys so a retried direct debit does not collect twice; failed collection follows a dunning schedule with backoff before escalation. Document integrity (an invoice PDF's hash stored with the ledger entry) uses Cryptographic hashes; synthetic customers for load tests come from a seeded PRNG.

Data and control flow

A monthly bill:

1. Meter reads for the period arrive over the month and land in the raw log; the validation stage writes cleaned intervals with quality flags.
2. On the cycle date the scheduler enqueues one job per account.
3. A worker leases a job, loads the account's facts and the tariff versions in force, and reads the intervals as of now.
4. Rating computes lines per tariff component with the money rule; the totals and taxes follow.
5. The worker posts the invoice to the ledger in one transaction (gapless number assigned inside it), with a fencing token and an idempotency key for the run.
6. A document is rendered and sent; a payment request is scheduled for the due date.
7. A failure at any step is retried from the last logged step; after repeated failure the job goes to the dead-letter queue.

A late corrected read:

1. The meter operator sends a corrected read for a period already billed.
2. Intake stores it as a new version and marks the old one superseded.
3. A change detector finds affected accounts by period overlap and enqueues re-rating.
4. Re-rating computes the bill as it should have been; the difference to the posted bill is posted as an adjustment, with a credit note or a debit line on the next bill.

Scaling

1. Billing window: a cycle must finish in hours. Levers: shard accounts across workers, keep the rater pure and in memory, pre-aggregate intervals per tariff band, spread cycles over the month.
2. Interval volume: data grows with meters times intervals per day. Levers: columnar compression, partition by month, move cold data to cheaper storage, store raw reads only once and derive the rest.
3. Ledger contention: popular accounts (large commercial customers) and the invoice number counter serialise writes. Levers: per-account versioning, allocate invoice numbers in blocks per legal entity where law allows, batch postings.
4. Retroactive changes: a tariff correction can touch every account. Levers: version tariffs, re-rate in bulk off-peak, limit the correction window by policy.
5. Customer-facing reads: portals and apps want usage charts. Levers: pre-aggregate hourly and daily series, cache.
6. Market calendars: month-end and year-end peaks. Levers: capacity for the peak, queue priorities.

Failure modes

Real implementations

The industrial energy CIS products (Kraken, SAP IS-U, Oracle, Gentrack) are closed; Octopus Energy publishes API documentation for its platform. None of the open projects above is an energy retail system: they cover the generic billing, metering and meter-reading seams.

Sources

• Wikipedia, Meter data management: import, validation and cleansing of meter data before billing. page
• Wikipedia, Time-of-use pricing: bands and tariffs. page
• Wikipedia, Customer information system. page
• Octopus Energy developer documentation (GraphQL and REST API). site
• Gurux.DLMS .NET library README: reading DLMS/COSEM meters. repo
• Kill Bill, Lago, OpenMeter, OpenLEADR repositories (links in the table above).