audio-codec

How it is built

Architecture overview

A real-time audio codec turns a stream of sampled sound into a few kilobits per second and back, fast enough for two people to talk over it. "Real time" sets the constraints: the codec works on short frames (milliseconds, not seconds), may look only a few milliseconds ahead, must produce an output for every frame even when a packet never arrived, and must run on a phone CPU alongside everything else. The one idea that makes it work is to spend bits only where the ear can tell: model the signal with a predictor or a transform, quantise the residual coarsely, and entropy-code what is left.

Two families do this. A linear-prediction (LP) coder treats speech as a filter excited by a source: it sends the filter coefficients and a quantised residual, which is very efficient for voiced speech at low rates. A transform coder turns each frame into frequency bands with a modified discrete cosine transform and quantises the bands, which handles music and mixed signals. Opus, the codec standardised in RFC 6716, contains both (the SILK layer and the CELT layer) and can run either alone or both together, so one format covers a voice call, a music stream and everything between (source: RFC 6716, section 2).

Around the core sit the pieces that make it survive a network: a decoder-side concealment of lost frames, optional redundant data for the previous frame, a voice activity detector that lets the sender go quiet, and resampling between the codec's internal rate and the sound card's.

Parts:

• Front end — frame the input, resample to the codec rate, detect speech, shape the spectrum — sample-rate-conversion (no page yet), IIR and FIR filters, Voice activity detection, Fast Fourier transform, biquad-filters (no page yet)
• Analysis and model — decide how many bits each part of the spectrum deserves, and predict the signal — psychoacoustic-model (no page yet), Linear predictive coding, modified-dct (no page yet)
• Quantisation — turn real-valued coefficients into small integers — vector-quantization (no page yet), scalar-quantization (no page yet)
• Entropy coding — pack the integers into the fewest bits — Arithmetic coding
• Loss handling — survive missing packets — packet-loss-concealment (no page yet), forward-error-correction (no page yet), recurrent-networks (no page yet)
• Bit-exact and fast implementation — same output on every CPU, quick on each — Deterministic fixed-point math, SIMD programming, PCG and xoshiro generators

 mic ─► capture ─► resample ─► frame (2.5 .. 60 ms) ─► VAD / mode & bandwidth decision
                                         │
                       ┌─────────────────┴──────────────────┐
                       ▼                                    ▼
              LP analysis (speech)                 MDCT + band energies (music)
              filter + residual                    energy ► shape vector
                       │ quantise                           │ quantise
                       └──────────► range coder ◄───────────┘
                                         │
                       packet (+ optional redundancy for the previous frame)
                                         │  RTP / UDP / Ogg
 speaker ◄─ resample ◄─ decode ◄─ jitter buffer ◄───── network (loss, delay, reorder)
                           ▲
                 concealment when a frame is missing

How the parts fit together

Front end: frames, rates and a speech detector

Everything downstream is organised by frame length. Opus allows frames of 2.5, 5, 10, 20, 40 or 60 ms and packets of up to 120 ms of audio; the RFC notes that more frames per packet means less header overhead but more delay and a bigger loss when one packet goes missing, and that 20 ms is a good choice for most uses (source: RFC 6716, section 2.1.4). The sound card rarely runs at the codec's rate, so a sample-rate-conversion (no page yet) stage sits on both sides; the MDCT layer in Opus always works internally at 48 kHz and the decoder just decimates for lower output rates (same source, section 2). A polyphase or windowed-sinc resampler is built from IIR and FIR filters and benefits from SIMD programming, because the same short dot product runs for every output sample.

The Voice activity detection block decides whether a frame carries speech. The sender uses it for discontinuous transmission, where in Opus "only one frame is encoded every 400 milliseconds" while the channel is idle (source: RFC 6716, section 2.1.9). Its contract with the encoder is a per-frame flag with hysteresis, so that word endings are not clipped. Spectral shaping and pre-emphasis use second-order sections, the biquad-filters (no page yet) block, because a cascade of them is cheap and stable.

Analysis: where the bits go

The psychoacoustic-model (no page yet) block is the classical answer to "which parts of the spectrum may be coded coarsely": masking thresholds computed from an Fast Fourier transform of the frame set the allowed quantisation noise per band. Low-delay codecs usually replace a full model with something cheaper: fixed band layouts, band energies sent explicitly, and noise shaping filters tuned for speech. Opus does the latter; its CELT layer codes the energy of each band first and then the shape within it, and its encoder section lists an energy-envelope quantiser and a spreading decision as separate steps (source: RFC 6716, sections 4.3 and 5.3). Treat the psychoacoustic model as "how the bit budget is split", not as a mandatory module.

For speech, Linear predictive coding predicts each sample from the previous ones, so the encoder sends the predictor and a small residual instead of the waveform. In Opus the LP layer is based on SILK and works at narrowband to wideband with frame sizes from 10 to 60 ms (source: RFC 6716, section 2). The MDCT layer (modified-dct, no page yet) uses overlapping windows so that the aliasing introduced by blocking cancels between neighbouring frames; the cost is a fixed look-ahead, 2.5 ms in CELT, which the RFC says cannot be reduced because the overlap size is fixed by the decoder (same source, section 2).

Quantisation and entropy coding: the contract between them

The quantiser outputs integers with known probability distributions; the entropy coder spends fractional bits on them. Opus uses the same range coder in both layers so there are no padding bits between the layers in hybrid mode (source: RFC 6716, section 2). Arithmetic coding is chosen over Huffman because a typical symbol here carries well under a bit of information (many bands quantise to zero), and a whole-bit code would waste most of it; the price is a slower, sequential decode that has to be bit-exact. A range coder is an arithmetic coder that works with integer ranges and byte output.

The RFC describes SILK as coding its excitation with a modified Pyramid Vector Quantiser, and CELT as coding the normalised shape of each band with a Pyramid Vector Quantiser (source: RFC 6716, sections 4.2 and 4.3). vector-quantization (no page yet) is the underlying idea: a codebook of whole vectors rather than independent values. Its advantage in a band-energy design is that the energy is already sent, so the shape vector always has unit norm and the coder cannot waste bits on level. scalar-quantization (no page yet) is the simple alternative per coefficient and is what you would pick for a first version of a new codec.

Loss handling: three layers

Packet networks lose frames, so the decoder must always produce something. The layers, from cheapest to most expensive:

1. Concealment at the decoder (packet-loss-concealment, no page yet). It is not part of the bitstream. In the Opus reference decoder the CELT path finds a periodicity in the decoded signal and repeats the windowed waveform with the pitch offset, while the SILK path extrapolates with the linear predictor (source: RFC 6716, section 4.4). It needs the same Linear predictive coding machinery as the encoder, which is why it is listed there.
2. Redundancy in the stream (forward-error-correction, no page yet). Opus can add a lower-bitrate re-encoding of a perceptually important previous frame (LBRR) to a later packet, so a single loss can be repaired from the next packet (source: RFC 6716, section 2.1.7 and 4.2). The sender raises or lowers this according to measured loss, because the redundancy costs bits when the network is clean.
3. Learned models (recurrent-networks, no page yet). Opus 1.5 added Deep Redundancy (DRED) and a Deep PLC to improve packet loss robustness (source: Opus 1.5 release note). Streaming models need small recurrent cells that keep state between frames, which is why GRU and LSTM cells are listed as the block.

The alternative for layer 3 is a transformer; for frame-by-frame streaming the recurrent cell is the natural fit because it carries a fixed-size state instead of attending over a growing history.

Bit-exactness and speed

A codec is a specification, and two implementations must decode the same bytes to the same samples, or close enough to be inaudible. In Opus the primary normative part is the reference source code in the RFC's appendix, and only its decoder portion is normative; the decoder contains integer and fixed-point arithmetic that must be performed exactly, rounding included (source: RFC 6716, section 1). Deterministic fixed-point math is the discipline behind that, and it is also what lets tests compare output byte for byte across compilers. For speed, the inner loops (filters, MDCT butterflies, pitch correlation) are written with SIMD programming. Any noise the decoder synthesises (comfort noise, noise filling) must come from a seeded generator such as those in PCG and xoshiro generators, so a given stream always decodes to the same output. (The Opus project itself ships hand-optimised AVX2 and Neon code, listed in its release notes.)

Data and control flow

One 20 ms frame, microphone to speaker:

1. The capture callback delivers 960 samples at 48 kHz (20 ms), resampled if the device runs at another rate.
2. The voice activity detector marks the frame as speech or silence; the mode logic picks LP, MDCT or hybrid and a bandwidth from the current bitrate target.
3. The encoder analyses the frame: LP coefficients and a pitch lag for speech, or band energies and a shape for music.
4. Coefficients are quantised and fed to the range coder; the result is for speech at 16 kbit/s, order of 40 bytes per 20 ms frame.
5. The packet is wrapped in RTP (the payload format is RFC 7587) and sent over UDP.
6. The receiver puts the packet in a jitter buffer that reorders and decides how long to wait.
7. The decoder decodes the frame; if it is missing, the concealment path synthesises one, or the redundancy in the next packet is used.
8. Resample, mix with other streams and hand to the sound card callback, which must never block.

One rate change under congestion:

1. The sender notices rising loss or delay from RTCP feedback.
2. The bitrate target is lowered; the encoder switches bandwidth (for example from fullband to wideband) and may enable redundancy.
3. The decoder follows the stream with no renegotiation, because Opus signals mode, bandwidth and frame size in each packet (source: RFC 6716, section 3).

Time goes to the pitch search and noise-shaping filters in the speech path, and to the transform plus the band quantiser in the music path. Opus is built so the algorithmic delay is between 5 ms and 65.2 ms depending on mode (source: RFC 6716, section 2); the delay the user hears is that plus device buffers and the jitter buffer, which are usually larger.

Scaling

The scarce resources are, in order:

1. Bits per second. The lever is the mode: LP at low rates for speech, MDCT for music, hybrid in between. The RFC gives sweet spots for 20 ms frames of 8-12 kbit/s for narrowband speech, 16-20 for wideband speech, 28-40 for fullband speech and 64-128 for fullband stereo music (source: RFC 6716, section 2.1.1).
2. Latency. Shorter frames and less look-ahead cost efficiency. Use 2.5-10 ms CELT frames for live music over a network, 20 ms for calls.
3. CPU on the phone. The encoder complexity setting trades CPU for quality, an integer from 0 to 10 in the reference encoder (source: RFC 6716, section 2.1.5). Lower it on weak devices, before touching bitrate.
4. Server cost in a group call. Decoding every stream to mix costs CPU per participant; forwarding without decoding moves the work to the clients.
5. Loss. Add redundancy and concealment; each costs bits or CPU.

Failure modes

Real implementations

The alternative: use libopus and do not write a codec. A new codec is only worth it for a special constraint (a few kbps over radio, a custom neural decoder, a fixed hardware block).

Sources

• J.-M. Valin, K. Vos, T. Terriberry, Definition of the Opus Audio Codec, RFC 6716, 2012. RFC
• RTP Payload Format for the Opus Speech and Audio Codec, RFC 7587. RFC
• Updates to the Opus Audio Codec, RFC 8251. RFC
• RTP: A Transport Protocol for Real-Time Applications, RFC 3550. RFC
• Opus project: site, libopus 1.5 release note
• Lyra README: github.com/google/lyra; Codec 2: github.com/drowe67/codec2
• Free Lossless Audio Codec format, xiph.org/flac/format.html