← Back
Home › Systems › Communication and media

music-royalty-collection

music usage matching and royalty distribution

GEMA, PRS, ASCAP

Open the music-royalty-collection pagemusic-royalty-collec… DAG workflow executiondistributed: Run a graph of dependent tasks by releasing each one when its inputs are done, longest chain first, and skip any task whose input hashes are unchanged.›Data shuffledistributed: Hash/range repartitioning of intermediate data between workers (exchange operator)Rights chain share resolutioneconomics: Resolve each party's royalty share by multiplying split percentages through the chain of rights holders, by territory and periodCurrency cross ratetriangulationfinance: Deriving cross rates through a base currency with correct precision, inversion and spreadsDouble-entry ledgerfinance: Append-only journal of transactions whose postings sum to zero per currency; balances are derived, so money can only move, never appear or vanish.›Largest remainder roundingfinance: Splitting an amount into parts that sum exactly (largest remainder / Hamilton) for allocations, splits and prorationsMoney representationfinance: Store money as integer minor units with an explicit currency exponent, compute in wider integers, and round once with a named rounding mode.›Reconciliationfinance: Match internal ledger records against bank and processor statements by reference, then amount and date window; every unmatched or mismatched item becomes an exception.›Text normalizationmachine-learning: Rewrite messy text into one canonical form for matchers and models, or into spoken words for speech output (and back), keeping a map to the original offsets.›Fuzzy string matchingsearch-sort: Edit distance, Jaro-Winkler and trigram indexes for approximate name and address matchingRecord linkagesearch-sort: Probabilistic record matching (Fellegi-Sunter) with blocking for patient or entity indexesBitemporal datastorage: Valid time plus transaction time on every record for as-of queries, back-dating and correctionsColumnar storagestorage: Store each column contiguously so analytic scans read only the columns they need, compress well and run as tight SIMD loops.›Entity resolutionstorage: Decide which records describe the same real-world thing: block to cut the pair space, score pairs with a probabilistic model, cluster the matches transitively.›Hash joinstorage: Build a hash table on the smaller input's join key and probe it with every row of the larger one; the linear-time workhorse for equi-joins.›Inverted indexstorage: Map each term to a sorted, compressed list of the documents containing it; the core of full-text search and other fast set queries.›Hierarchical state machinesystems: Nest states so a substate inherits its parent's reactions and only overrides what differs, with ordered entry and exit actions along the path.›Rete rules enginesystems: Match many rules against changing facts by caching partial matches in a shared network, so each change only does the work it actually affects.›Durable job queuedistributed: Persisted tasks that workers lease for a visibility timeout, ack on success and retry with backoff; at-least-once execution that survives worker and queue crashes.›Topological sortgraphs: Order the vertices of a DAG so every edge points forward; the core of build systems, schedulers and dependency resolution, with cycle detection for free.›Content-addressed storagestorage: Store each blob under the hash of its bytes: immutable, deduplicated by construction, self-verifying on every read, and composable into Merkle trees.›Exponential backoff with jittersystems: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.›Fast hash functionsdata-structures: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.›Range partitioningdistributed: Cut the sorted key space into contiguous half-open ranges kept in a boundary map; split hot or big ranges, merge small ones, move ranges between nodes.›External merge sortsearch-sort: Sort data larger than RAM: write memory-sized sorted runs, then k-way merge them with large sequential buffers; terabytes in one merge pass.›Columnar memory formatstorage: In-memory column buffers with validity bitmaps and offsets (Arrow style) for zero-copy exchangeIdempotency keysfinance: Client-chosen unique key per logical operation, stored with a request fingerprint and the final response, so retries replay the first result instead of acting twice.›Linear and logistic regressionmachine-learning: Fit a weighted sum of features to a number (least squares) or a probability (logistic); the baseline every model must beat.›Near-duplicate detection withMinHash LSHmachine-learning: Estimate set Jaccard similarity from k min-hashes and find similar pairs with banded LSH buckets; the standard dedup for web crawls, training corpora and sequence collections.›Interval treedata-structures: Index of ranges that answers "which intervals overlap this point or range?" in O(log n + k); static sorted-array form is the compact, cache-friendly choice.›+ 14 moreDead-letter queuedistributed: A side queue for messages that keep failing: after N delivery attempts or a permanent error, park them with the reason so the main queue keeps moving.›Write-ahead logstorage: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›BFS and DFSgraphs: The two basic graph traversals; BFS gives hop distances and levels, DFS gives discovery/finish order for cycles, components and topological order.›Merkle dagdata-structures: Content-addressed DAG of hashed nodes (git trees/commits) for dedupe and verifiable historyAtomic file replacementhardware-os: Write temp file, fsync, rename over target, fsync directory: crash-safe whole-file updatesPRNGs (PCG, xoshiro)numerics: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.›Leasesdistributed: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.›Cryptographic hashes (SHA-2,BLAKE3)security: Collision-resistant fingerprints of data; SHA-256 for interoperability, BLAKE3 for speed, HMAC/HKDF for keyed uses.›Buffer pools forzero-allocation I/Ohardware-os: Preallocate a fixed set of aligned I/O buffers and recycle them through a free list, so the hot path never mallocs, faults or unmaps. The bounded count doubles as back pressure.›K way mergesearch-sort: Merge k sorted streams with a heap or loser tree (merging iterators)Pattern-defeating quicksort(pdqsort)search-sort: Introsort successor: quicksort with adaptive pivots, linear on sorted and equal-heavy input, heapsort fallback; the unstable sort of Go and (until 1.81) Rust.›Loss functionsmachine-learning: The scalar a model minimizes: cross-entropy for classes and tokens, MSE or Huber for values, each with a stable form and simple gradient.›+ 8 moreCRC32C checksumshardware-os: The Castagnoli CRC-32 with dedicated instructions on x86 (SSE4.2) and ARMv8. It is the standard fast integrity check for storage blocks, logs and network frames.›Sparse matrices (CSR)numerics: Compressed sparse row stores only nonzeros as row pointers, column indices and values; SpMV streams them once, so it is memory-bandwidth bound.›fsync, O_DSYNC and O_DIRECThardware-os: What fsync, fdatasync, O_DSYNC, O_DIRECT and F_FULLFSYNC really guarantee on Linux and macOS, and the exact call sequences for crash-safe files.›Fencing tokensdistributed: A number that grows with every lock grant and is checked by the protected resource, so a paused ex-holder's late write is rejected.›Memory orderingconcurrency: What Relaxed, Acquire, Release and SeqCst atomics actually guarantee, what they cost on x86 and ARM, and which one each pattern needs.›+ 1 more

How it is built

Architecture overview

A collecting society (GEMA in Germany, PRS in the UK, ASCAP in the US) licenses the use of its members' compositions and pays the money back to the right people. CISAC, the confederation of such societies, describes itself as a network of 225 societies in 110 countries (source: CISAC website). The software problem is a very large join under uncertainty. On one side is a repertoire: very many musical works, each with a chain of writers and publishers and their percentage shares, which change over time. On the other side is usage: reports from streaming services, broadcasters, venues and radio monitoring, with very many lines in which a work is named only by a title, an artist and perhaps a recording code. The system matches each usage line to a work, works out who is owed what under the licence and territory rules, rounds and converts money so that the parts add up exactly, and pays out through a ledger that can be reconciled.

The idea that makes it work is that money is computed only after identity is settled, and identity is a confidence-scored match, not a lookup. Exact identifiers (ISRC for a recording, ISWC for a work, IPI for a rights holder) resolve the majority of lines cheaply; probabilistic matching handles the rest; a human queue handles what remains. The share arithmetic is then deterministic integer math over a bitemporal registry, which makes a distribution reproducible and a correction possible after the fact.

Parts:

 members ─► register works, shares ─► repertoire registry (bitemporal)
                                            │ as-of lookups
 services, broadcasters, venues, radio       ▼
   ─► usage files (DDEX DSR, CSV) ─► ingest ─► normalise ─► match ──► matched lines
                                            │       │ exact codes (ISRC, ISWC)   │
                                            │       └ fuzzy + model ► review queue
                                            ▼                                    ▼
                                      rules (licence, tariff, territory) ► distribution (shares)
                                                                                 │ integer money
                                                                                 ▼
                              reconcile ◄── ledger (double entry) ◄── statements ► payouts

How the parts fit together

Repertoire registry

Every distribution is "who owned what share of which work on the day of use". A work has writers and publishers with shares that must add up to the whole; shares change when a contract ends or a dispute is settled, often retroactively. The registry therefore keeps two time axes: when the fact is true in the world and when the society learned it (bitemporal-data, no page yet). A distribution run asks for the repertoire as of a date and as known on a date, which is what makes a corrected run reproducible. MVCC gives consistent snapshots to the long-running job while members keep editing, an Interval tree answers "which agreements cover this date", and Time partitioning keeps old periods out of the hot path. Standards carry the same data between societies: DDEX publishes a Musical Work Data and Rights message suite and a Recording Data and Rights suite, among others (source: DDEX standards page). A Content-addressed storage layer for each registered document, hashed with Cryptographic hashes, proves later which version of a split sheet was used (merkle-dag, no page yet, covers chaining versions).

Identity and normalisation

Titles arrive as "Song (feat. X) - Remastered 2011", with different Unicode forms of the same accent, quote marks and punctuation. Text normalization maps each to a canonical form before anything is compared, and Unicode text tokenization splits multilingual text into index terms. Codes are checked, not trusted: an identifier with a bad check digit or length is flagged at ingest. Fast hash functions fingerprint normalised keys for joins and dedupe.

Identifier families matter more than any algorithm. MusicBrainz, an open database, lists ISRC on recordings, ISWC on works and IPI on artists and labels as fields (source: MusicBrainz Database documentation). ISRC (recording) and ISWC (work) are ISO standards, ISO 3901 and ISO 15707, cited by number only; IPI is listed among the CISAC services (source: CISAC website). The chain recording to work is the first lookup: a service reports an ISRC, the registry maps it to a recording, and the recording to the work it performs.

Usage ingest

Usage files are large, regular and tabular. DDEX defines a Digital Sales Reporting message suite for exactly this report (source: DDEX standards page); a society also receives broadcaster cue sheets and spreadsheets. Ingest writes each file atomically (atomic-file-replacement, no page yet), validates it, and stores lines in Columnar storage, because matching and aggregation touch few columns of many rows. Block-level min/max and dictionaries (zone-maps, dictionary-encoding, both no page yet) skip work, and Vectorized execution processes a column at a time. Files bigger than memory are sorted by External merge sort with a k-way-merge (no page yet) of runs, so that lines for the same work sit next to each other. A CRC-32C checksums value per file part detects a truncated transfer before a month of money depends on it.

Matching: from a line to a work

Matching has three tiers. Exact codes: a hash lookup from ISRC or ISWC to a work, handled with a Swiss table in memory. Text search: the title and artist are searched in an Inverted index of work titles and writers; candidate lists come from Posting-list intersection and are ranked by BM25 ranking, keeping the best few with Top-k selection. Probabilistic linkage: the formal framework is Fellegi and Sunter's, which treats matching as a statistical decision over field-by-field comparisons (Fellegi and Sunter, A Theory for Record Linkage, JASA, 1969; record-linkage, no page yet, with fuzzy-string-matching, no page yet). Entity resolution generalises it to clustering of records into entities; Union-find merges pairwise matches into groups, Near-duplicate detection with MinHash generates candidate pairs without comparing all of them, and Embeddings can represent a title and artist as a vector for nearest-neighbour search. A trained scorer (a regression or small network) combines these signals; its calibration decides the threshold between automatic match and human review.

Lines with no usable text or code, typical for radio and venue sound, are identified from the audio. Wang's audio search algorithm uses a combinatorially hashed time-frequency constellation, identifying a short segment against a database of over a million tracks, with search times of milliseconds per query (Wang, An Industrial-Strength Audio Search Algorithm, ISMIR 2003). Chromaprint, the AcoustID client library, extracts fingerprints from any audio source (source: AcoustID).

Rules

A usage line becomes a payable item only under a licence: a service's blanket licence, a broadcaster's tariff, a venue's per-event fee. Rules combine territory, rights type (performing, mechanical), usage type and dates, and they change by agreement. A Rete rules engine matches many rules against the facts of a line without re-evaluating everything for each line; a Hierarchical state machine tracks a claim or dispute through states with inherited transitions (registered, matched, disputed, held, paid). The United States split is an example of how rights types separate: The MLC administers blanket mechanical licences for eligible streaming and download services in the US, and collects and pays mechanical royalties (source: The MLC website).

Distribution engine

For each period the engine joins matched lines with the registry: work to parties, parties to shares, minus society deductions. It is a Hash join on work identifier followed by a group-by on party, spread over workers by Range partitioning or a shuffle (data-shuffle, no page yet). An Arena allocator holds the per-run structures and frees them at the end of the run. Where a work has several rights holders, shares multiply down the chain: a publisher's share of a writer's share times the work's share of the line (rights-chain-share-resolution, no page yet). The result must be exactly the amount received, so allocation uses largest-remainder-rounding (no page yet) on integer amounts.

Money

Amounts are stored as integers in minor units with an explicit currency (Money representation). Foreign usage is converted with a rate set per period; cross rates are derived through a base currency with a defined rounding order (currency-cross-rate-triangulation, no page yet). The ledger is an append-only Double-entry ledger: each distribution run posts balanced entries (receipts, deductions, party balances, payouts), so the total of all party balances plus the society's own always equals what was received. Reconciliation compares the ledger with bank statements and with the reports from other societies, which exchange money under reciprocal agreements.

Workflow and reliability

A monthly or quarterly run is a graph of jobs: ingest, normalise, match, price, distribute, post, export statements. A DAG workflow execution engine releases a job when its inputs exist; jobs are leased from a Durable job queue, retried with Exponential backoff with jitter, and moved to a Dead-letter queue after repeated failure. Every job step has an Idempotency keys value so that a retried job posts to the ledger once. A Leases value keeps two workers from running the same partition, and Fencing tokens make the ledger reject a write from a worker that lost its lease. The run state sits behind a Write-ahead log, and final files are made durable with fsync and direct I/O.

Data and control flow

One quarterly distribution for a streaming service:

  1. The service delivers a usage report (a DDEX DSR file or a CSV). Ingest checks checksum and schema and stores the file under its content hash.
  2. Normalisation produces canonical title, artist and codes; lines are written to columnar storage grouped by territory.
  3. Matching: codes first, then text search, then probabilistic scoring. Matched lines carry a confidence; lines under the threshold go to the review queue and are held, not paid.
  4. The rules engine assigns the licence, rights type and rate; an unlicensed line is parked with a reason.
  5. The distribution engine reads the registry as of the usage date, resolves shares down the rights chain, applies deductions and rounds with a largest-remainder rule.
  6. Posting: one balanced ledger transaction per party per period, keyed by run and party so a retry is a no-op.
  7. Statements and payouts are generated, bank files are exported, and reconciliation closes the period.

A late correction:

  1. A member's share on a work is corrected retroactively; the registry records the new fact with the date it became known.
  2. A job selects affected past lines by work and date range (interval lookup).
  3. It recomputes only the difference against the earlier distribution and posts adjusting entries in the current period; the past ledger is never edited.

Scaling

In the order you hit them:

  1. Matching volume. Very many lines, mostly repeats of the same recordings. Lever: deduplicate by normalised key first and match each unique key once, then fan the result back out with a hash join.
  2. Text-search quality. More recall brings more review work. Lever: tighter blocking, a calibrated model, active learning on reviewer decisions.
  3. Join and group-by size. Lever: partition by work, columnar scans, vectorised operators, SIMD for the hot loops.
  4. Registry hot spots. Popular works have many parties. Lever: cache share chains per work and date, invalidate on registry change.
  5. Run latency. A late file delays payout. Lever: DAG parallelism, per-territory partitions, incremental runs.
  6. Review queue. Humans are the slowest part. Lever: rank by amount, auto-accept above a confidence, report match rate by source to the service.

Failure modes

FailureSymptomCauseFix
Wrong work matchedroyalties paid to the wrong writersfuzzy match above threshold on a common titlecalibrated threshold, check artist and duration, dispute process with clawback
Unmatched usagelarge "unidentified" poolmissing codes, new works not yet registeredISRC to work table from labels, audio fingerprinting, hold for later matching
Shares do not sum to 100 %distribution rejected or leaks moneydata entry errors, overlapping agreementsvalidate at registration; block activation of an invalid chain
Rounding driftcents missing between receipts and payoutsindependent rounding per partylargest-remainder allocation; ledger check that debits equal credits
Double postingparty paid twicejob retried after a timeoutidempotency key per run, party and period; fencing on the ledger writer
Late data changes historya closed period no longer matchesretroactive registry edit overwrote old databitemporal records; post adjustments in the current period
Currency mismatchbalances in the wrong currencyimplicit currency, wrong rate datemoney type with explicit currency; rate table with effective dates
Truncated usage filemonth missing for one serviceinterrupted transferchecksum and line-count control totals before ingest
Duplicate usage fileusage counted twicefile resentcontent hash on ingest; reject known hashes
Disputed work frozenmoney held for longcompeting claims on the same workhold in suspense with an age limit; escalate to the society's committee

Real implementations

namewhat it isrepolicencenotes
MusicBrainzopen music metadata database with recordings, works, ISRC and ISWC fieldsmetabrainz/musicbrainz-serverGPL-2.0-or-later (server; data CC0 for core data per the database docs)a reference for identifier relations, not a royalty system
Chromaprint / AcoustIDaudio fingerprint library and serviceacoustid/chromaprintMIT own code; whole build LGPL-2.1 because of bundled FFmpeg parts (per LICENSE.md)recording identification
Splinkprobabilistic record linkage in Pythonmoj-analytical-services/splinkMITprobabilistic record linkage; term-frequency adjustments
dedupemachine-learning fuzzy matching and entity resolutiondedupeio/dedupeMITPython library

The alternative: a small society or a publisher with a handful of catalogues can run on spreadsheets, a relational database and a few scripts, with human review of every unmatched line. A purpose-built engine pays off when usage files reach many millions of lines and the pending queue costs more than the software.

Sources

Similar systems

Share the most building blocks with this one.

procurement 26 %securities-custody 26 %insurance-policy-admin 24 %accounting 22 %title-escrow 22 %