← Back
Home › Systems › Communication and media

video-call

Turns each participant's camera and microphone into a live stream and delivers it to every other participant within a few hundred milliseconds through a selective forwarding server.

WebRTC SFU, Zoom

Open the video-call pagevideo-call Hdr histogramdata-structures: Log-linear bucketed histograms with bounded relative error for latency percentilesConsistent hashingdistributed: Hash nodes and keys onto a ring; each key belongs to the next node clockwise, so membership changes move only about 1/n of keys.›Deterministic simulationtestingdistributed: Run the whole system in one thread under a seeded scheduler with simulated network, disk and clock, so every failure replays from its seed.›Primary-backup replicationdistributed: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›Process supervisionhardware-os: A parent process starts services, reaps their exits and restarts them under a delay and burst-limit policy, then stops them with SIGTERM before SIGKILL.›Voice activity detectionmachine-learning: Per-frame speech/non-speech decision with smoothing, from an energy threshold over a tracked noise floor up to small neural nets.›Acoustic echo cancellationmedia: Adaptive filter (NLMS/frequency-domain) removing far-end echo from the microphone signalA/V sync and media clocksmedia: How audio and video stay lip-synced: per-stream timebases, PTS/DTS, PCR and RTCP clock mapping, and a master clock that slave streams chase by dropping, repeating or resampling.›Hardware video accelerationmedia: GPU/ASIC encode and decode APIs (NVENC, VideoToolbox, VA-API) with zero-copy surfacesJitter buffermedia: A receiver-side buffer holds packets just long enough, a delay tracked from recent arrivals, to reorder them and hand out media at a steady rate.›NAT traversal (STUN, ICE, TURN)media: Learn your public address with STUN, swap candidate addresses over signalling, test pairs with ICE checks, and fall back to a TURN relay if no direct path works.›Noise suppressionmedia: Spectral gain estimation (Wiener/RNN) to remove background noise from speechPacket loss concealmentmedia: Synthesize missing audio frames from prior signal (pitch repetition, neural PLC)RTP and RTCPmedia: RTP stamps each media packet with sequence number, media-clock timestamp and source ID; RTCP reports loss, jitter, round trip and clock mapping within a bandwidth cap.›Sdp offer answermedia: SDP session descriptions and offer/answer negotiationSelective forwarding unitmedia: A server receives each participant's media once and forwards chosen packets to every other participant, picking the quality layer per receiver without decoding anything.›Simulcast svcmedia: Simulcast and scalable video coding layers for per-receiver qualitySRTPmedia: Per-packet encryption, authentication and replay protection for RTP and RTCP that keeps the headers readable, with the packet index derived from the 16-bit sequence number.›Video rate controlmedia: CBR/VBR/CRF rate control with VBV buffer model and lookaheadDelay based congestion controlnetworking: One-way delay gradient estimation for real-time media (GCC, transport-cc)WebSocketsnetworking: Full-duplex message framing over one upgraded HTTP connection; the standard way to push real-time data to browsers.›Coverage-guided fuzzingsecurity: Mutate inputs, run an instrumented target, and keep any input that reaches a new code edge; feedback turns blind guessing into incremental search.›Mutual TLSsecurity: TLS where the client also proves a certificate key, so every connection carries a verified, rotatable machine identity.›Forward error correctionsignal-processing: Add redundancy (LDPC, Reed-Solomon, XOR parity) so receivers repair loss without retransmitPoint-in-time recoverystorage: Periodic base backups plus a continuously archived change log let you rebuild the state as of any instant, then branch from there.›Graceful drainingsystems: On a termination signal, stop taking new work, finish in-flight work until a deadline, tell clients to reconnect elsewhere, then exit; clients resume from a token.›Cooperative schedulingconcurrency: Green threads/coroutines yielding at await points on a user-space schedulerFault injectiondistributed: Inject latency, errors, kills and partitions via proxies, netem or hooksRecord replaydistributed: Record every nondeterministic input (time, randomness, I/O, model and tool outputs, per-tick player input) and re-execute deterministically for debugging, replays and state rebuildLeasesdistributed: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.›Health checkingsystems: Decide per backend whether to route to it using probes and real-traffic errors, with consecutive-count hysteresis, capped ejection and a panic fallback.›Realtime audio callbackmedia: Allocation- and lock-free audio thread design with bounded work per bufferFast Fourier transformnumerics: Computes the DFT in O(n log n) by recursively splitting into even and odd halves; the basis of fast convolution and spectral methods.›IIR and FIR filterssignal-processing: Two ways to shape a sampled signal's spectrum: FIR (finite taps, always stable, exact linear phase) or IIR (feedback, a few coefficients, cheap but phase-distorting).›Media containersmedia: MP4/ISO BMFF, fragmented MP4/CMAF, Matroska and MPEG-TS mux/demuxParameter smoothingmedia: Per-sample ramps of control parameters to avoid zipper noiseNetwork address translationnetworking: NAT mapping tables, endpoint-independent mapping/filtering, hairpinning and port allocationMessage authentication codes(HMAC, CMAC, Poly1305, KMAC)security: A keyed short tag proving a message is unmodified and came from a key holder; pick HMAC, CMAC, KMAC or a one-time polynomial MAC by what primitive you already have.›+ 19 moreTransparent interceptionnetworking: Redirect traffic to a local proxy (iptables TPROXY/REDIRECT, eBPF) without app changesSyscall interceptionsecurity: Intercept syscalls via ptrace, seccomp user notify or a user-space kernelFencing tokensdistributed: A number that grows with every lock grant and is checked by the protected resource, so a paused ex-holder's late write is rejected.›Biquad filterssignal-processing: Second-order IIR sections (RBJ cookbook): low/high pass, peaking EQ; stable cascades+ 9 moreEbpfhardware-os: Verified in-kernel programs on hooks (kprobes, XDP, tc) with mapsPtrace process controlhardware-os: Ptrace: attach, read and write registers and memory, single-step, intercept signalsSeccomp bpfsecurity: Syscall filtering with seccomp BPF programs+ 6 more+ 1 more

How it is built

Architecture overview

A video call turns every participant's camera and microphone into a live stream and delivers each stream to every other participant within a few hundred milliseconds. The design that scales beyond two or three people is the selective forwarding unit (SFU): each client sends its media once to a server, and the server forwards chosen packets to each receiver without decoding or mixing them. The one idea that makes it work is that quality is decided per receiver and per moment: the sender uploads several versions of its video (layers), the server picks one layer for each receiver from that receiver's measured bandwidth, and a separate, lightweight signalling channel sets everything up. Everything else (NAT traversal, encryption, loss repair, clock sync) exists to keep this loop alive on bad networks. WebRTC is the browser standard for the client side; Zoom, Meet and Teams use proprietary variants of the same architecture.

Parts:

 Client A (browser/app)                                          Client B
 mic ─► echo cancel ─► noise supp ─► Opus ─┐                 ┌─► jitter buf ─► decode ─► speaker
 cam ─► encode 3 layers (simulcast) ───────┤                 │   + PLC, A/V sync
      ▲ target bitrate                     ▼                 │
      └──── congestion control ◄── RTCP ── SFU (media server) ─┘  picks 1 layer per receiver
                                            ▲   │
   signalling (WebSocket, SDP, ICE cands)   │   └─► recording worker ─► object store
 ─────────────── room service ──────────────┘
        STUN/TURN: address discovery and relay fallback

There are two planes. The signalling plane is ordinary request/response over a WebSocket and tolerates a second of latency. The media plane is UDP packets with a budget of tens of milliseconds per hop, and it never goes through the signalling server.

How the parts fit together

Signalling

Signalling carries the session description and the room events, not the media. Each client opens a WebSocket to the room service, authenticates (a signed join token whose integrity is a message authentication code), and exchanges descriptions of what it will send and receive. Descriptions use the SDP offer/answer model (sdp-offer-answer, no page yet): one side offers codecs and transport parameters, the other answers with the subset it accepts (RFC 3264, RFC 8829). Why a WebSocket and not HTTP polling: the server must push events (someone joined, a stream was added) without a request. The signalling server owns room membership; it does not see media. Reconnects use exponential backoff with jitter so that a server restart does not bring every client back in the same second.

Connectivity

Most clients are behind NAT (network-address-translation, no page yet), so the server cannot simply send packets to them. NAT traversal combines three tools: STUN to learn the public address, ICE to try candidate address pairs and pick one that works, and TURN to relay everything through a server when no direct path exists (RFC 8445, RFC 8489, RFC 8656). An SFU is usually on a public address, so for client-to-SFU the client's checks almost always succeed without a relay; TURN is the fallback for clients behind strict firewalls, and it costs the operator relay bandwidth. Candidates are sent over signalling, which is the contract between this part and the previous one. Failed checks are retried with backoff rather than in a tight loop.

Media transport

Audio and video travel as RTP packets over UDP: each packet carries a sequence number, a media-clock timestamp and a source ID. RTCP rides alongside and carries reports of loss, jitter and round-trip time, capped at a fraction of the session bandwidth (RFC 3550 sets the default RTCP share to 5 % of session bandwidth). Every packet is encrypted and authenticated with SRTP while headers stay readable, so the SFU can route by header without holding the media keys when end-to-end encryption is used. In WebRTC, the SRTP keys come from a DTLS handshake on the same path. UDP is chosen over TCP because a retransmitted packet that arrives 300 ms late is useless; the receiver prefers to conceal the loss.

Forwarding (the SFU)

A selective forwarding unit is the alternative to two other designs. In a mesh every client uploads to every other client; upload grows with participants and breaks around a handful of people. An MCU decodes and mixes everyone into one stream; receivers need little bandwidth but the server pays heavy CPU for decoding and encoding, and every receiver gets the same composite. The SFU sits between: the server's cost is network and packet rewriting, not codec work. The contract with senders is simulcast or scalable video coding (simulcast-svc, no page yet): the sender encodes two or three resolutions of the same camera at once, each tagged so the SFU can recognise it. The contract with receivers is that the SFU forwards exactly one layer per sender and rewrites sequence numbers and timestamps so the receiver sees a continuous stream when the layer switches. A layer switch needs a keyframe, which the SFU requests from the sender through RTCP. Active-speaker detection (using voice activity signals) decides who is shown large and which streams are forwarded at all in a large meeting.

Sender encoding

The sender picks the encoder settings from the target bitrate the congestion controller reports. Video codecs (video-codecs, no page yet) compress with inter-frame prediction, which makes the GOP structure a latency knob: B-frames and long GOPs save bits but add delay, so conferencing encoders use low-delay configurations with few or no B-frames, and rely on requested keyframes to recover. Rate control (video-rate-control, no page yet) must hold the bitrate close to the target every second, not just on average, because the network queue sees bursts. Adaptive quantization spends bits where the eye notices, and perceptual quality metrics are used offline to tune encoder settings. Hardware encoders (hardware-video-acceleration, no page yet) are preferred on phones for battery.

Congestion control

Sending at the wrong rate is the main cause of bad calls. Delay-based congestion control (delay-based-congestion-control, no page yet) watches how packet arrival times drift: a growing one-way delay means a queue is filling somewhere on the path, so the rate is cut before packets are dropped. Google's published controller has two parts, a delay-based and a loss-based controller, and the delay-based part estimates the delay gradient with a recursive filter in the family of the Kalman filter (draft-ietf-rmcat-gcc describes the delay-based and loss-based controllers). Estimates are smoothed (exponential smoothing) so one noisy sample does not make the picture oscillate. The SFU adds a second control loop: it estimates each downlink and chooses the layer to fit. Bandwidth probing, which briefly sends extra packets to test headroom, is how the estimate grows after a drop.

Receiver playout

The receiver puts packets into a jitter buffer that waits just long enough to reorder them and absorb arrival-time variation; the delay target follows recent jitter, trading latency for smoothness. For audio the buffer is adaptive and stretches or compresses sound slightly rather than dropping it. Lost audio frames are filled by packet-loss concealment (packet-loss-concealment, no page yet): classic codecs extrapolate the pitch cycle using linear prediction, newer systems use small recurrent networks (recurrent-networks, no page yet). Redundancy such as forward error correction (forward-error-correction, no page yet) can repair loss without a retransmission, and for video a negative acknowledgement asks for a resend only when the round trip is short enough. Audio and video travel as separate streams with separate clocks; A/V sync clocks uses the RTCP mapping between each stream's media clock and wall-clock time to hold lip-sync.

Audio front end

The microphone hears the loudspeaker. Acoustic echo cancellation (acoustic-echo-cancellation, no page yet) models the room with an adaptive filter fed by the far-end signal and subtracts the predicted echo. Filters (FIR and IIR) and frequency-domain processing (FFT, STFT) do the work in frames of a few milliseconds. Noise suppression (noise-suppression, no page yet) estimates a gain per frequency band. All of it runs in a real-time audio callback (realtime-audio-callback, no page yet) that may not allocate memory or take locks. The output goes to an audio codec; Opus is the mandatory WebRTC codec (RFC 7874).

Room placement and operations

A room should live on one SFU so its streams are forwarded in-process. Placement uses consistent hashing over healthy nodes or a lease-based registry (leases): a node owns a room while its lease is current. Health checks remove sick nodes, and graceful draining lets a node leave on deploy by refusing new rooms while existing calls end, or by asking clients to reconnect elsewhere. A supervisor restarts crashed media processes. Control traffic between the room service and the media nodes uses mutual TLS. Large meetings cascade: SFUs forward to each other so each region's participants talk to a nearby node.

Each media node records join time, one-way media delay, jitter and the time to forward a packet in hdr-histogram (no page yet) histograms. Call quality is decided by the slow tail, since one participant with bad percentiles is a user with a bad call. Placement and draining read these percentiles: a node whose tail grows stops receiving new rooms before its health check fails.

Recording and state

Recordings are written by a worker that joins the room as a hidden participant, muxes the streams into a container (media-containers, no page yet) and uploads to object storage. Room and account metadata sits in a database with primary-backup replication, whose write-ahead log also enables point-in-time recovery. Live media state is deliberately not persisted: if an SFU dies, clients reconnect and renegotiate.

Verification

Packet parsers and jitter logic are the code that sees hostile input, so they are fuzzed (coverage-guided fuzzing). Network behaviour (loss, reordering, partitions, slow links) is tested under a seeded simulator (deterministic simulation testing), and in staging with injected faults (fault-injection, no page yet).

Data and control flow

Joining a call and sending video (one participant):

  1. The client calls the room API; the room service authenticates it, picks an SFU (hash or lease) and returns a token and addresses. Reads: room registry. Writes: membership.
  2. The client opens the WebSocket and sends an SDP offer; the server answers. ICE candidates flow both ways.
  3. ICE checks run on candidate pairs; the first working pair is nominated, or traffic falls back to TURN. Time here is a few round trips: this is the "call setup" delay people feel.
  4. DTLS handshake derives SRTP keys; media starts.
  5. The sender encodes three simulcast layers; each is packetised, encrypted and sent. RTCP sender reports map media clocks to wall-clock time.
  6. The SFU reads RTP headers, picks a layer per receiver from that receiver's bandwidth estimate and active-speaker state, rewrites sequence numbers and forwards. No decoding happens, so per-stream CPU is low.
  7. The receiver buffers, decodes, conceals loss and renders. Receiver reports (loss, jitter) return through the SFU to the sender's congestion controller. Time budget: the network path, the jitter buffer, codec framing and encode/decode all share the end-to-end delay; the jitter buffer is the part the receiver can trade away (order of tens of milliseconds).

Network degrades mid-call:

  1. Receiver reports show rising delay; the downlink estimate falls.
  2. The SFU switches that receiver to a lower layer at the next keyframe, requesting one from the sender.
  3. If the sender's own uplink is the problem, its estimate falls and it drops the top layer.
  4. As delay settles, probing raises the estimate and the layer steps back up.

Server failure:

  1. Health checks fail or the lease expires; the room is reassigned.
  2. Clients detect lost media, reconnect over signalling with backoff and jitter, renegotiate and resume.

Scaling

In the order they bite:

  1. Uplink and CPU of one SFU. Forwarding is cheap per stream but fan-out is quadratic in a room (every participant receives from many senders). Levers: forward only the active speakers and a limited number of thumbnails per receiver, lower layers for small tiles, more SFU cores with one thread per room shard.
  2. Rooms per node. Rooms are independent, so add nodes and place rooms by hash or lease. A single very large meeting (webinar) is the hard case: cascade SFUs or send viewers a one-way stream by CDN.
  3. Cross-region latency. Participants in different continents should each connect to a nearby SFU, with the SFUs linked on a backbone.
  4. TURN relay bandwidth. Relayed traffic is paid twice. Lever: more public SFU addresses and TCP/TLS-443 fall-back only when necessary.
  5. Recording and transcoding. Mixed recordings and MCU-style composites decode and re-encode and cost far more CPU than forwarding; run them on separate workers.
  6. Client CPU and battery. Sending three layers costs encoder time; hardware encoders and lower layer counts help.

Failure modes

FailureSymptomCauseFix
ICE failscall never starts, or only for some userssymmetric NAT or blocked UDP, no TURNprovide TURN over TCP/TLS 443; log candidate-pair stats
Bufferbloataudio delay grows, then drops outsender ignores delay signals or probes too harddelay-based control; pace packets
Keyframe stormvideo freezes, bandwidth spikesmany receivers request keyframes at oncerate-limit keyframe requests; periodic keyframes; SVC
Layer flappingpicture quality oscillatesnoisy bandwidth estimates, no hysteresissmooth the estimate, wait before upgrading
Echo or howlingfar end hears itselfecho canceller diverged, wrong audio delay estimatereset the adaptive filter; headphones prompt
Lip-sync driftaudio and video separate over timeclocks not mapped through RTCPuse sender reports to align both streams
Thundering herd on reconnectserver overloaded after a restartall clients reconnect at oncebackoff with jitter; admission control
SFU crashwhole room dropsmedia state lives only in one processfast detection, reassign room, clients auto-rejoin
Replay or forged packetsgarbled media, injected audiono packet authentication or key reuseSRTP with replay protection; rotate keys

Real implementations

namewhat it isrepolicencenotes
LiveKitopen-source SFU with client SDKs and cloud optionlivekit/livekitApache-2.0Go, built on Pion; rooms, simulcast, cascading
mediasoupSFU library for Node.js/Rust with a C++ workerversatica/mediasoupISCyou write the signalling yourself
Janusgeneral WebRTC server with pluginsmeetecho/janus-gatewayGPL-3.0plugins include a video-room SFU
Jitsi VideobridgeSFU of the Jitsi Meet stackjitsi/jitsi-videobridgeApache-2.0Java; part of a larger Jitsi stack
PionWebRTC stack in Go, used to build serverspion/webrtcMITlibrary, not a product
libwebrtcthe browser implementation: congestion control, jitter buffer, echo cancellerwebrtc.googlesource.com/srcsee reporeference for the client side

The alternatives: a full mesh (peer to peer) needs no server media cost but works only for very small groups; an MCU mixes streams on the server, which suits legacy room systems and constrained receivers but costs CPU and removes per-receiver quality choice. For broadcast-style meetings with thousands of passive viewers, an HTTP-based stream via a CDN (CDN and edge) trades 5-30 s of latency for scale; see Adaptive bitrate streaming.

Sources

  • RFC 3550, RTP: A Transport Protocol for Real-Time Applications (RTCP bandwidth share). rfc
  • RFC 3711, The Secure Real-time Transport Protocol (SRTP). rfc
  • RFC 8445, Interactive Connectivity Establishment (ICE). rfc
  • RFC 8489, Session Traversal Utilities for NAT (STUN). rfc
  • RFC 8656, Traversal Using Relays around NAT (TURN). rfc
  • RFC 3264, An Offer/Answer Model with SDP. rfc
  • RFC 8829, JavaScript Session Establishment Protocol (JSEP). rfc
  • RFC 8834, Media Transport and Use of RTP in WebRTC. rfc
  • RFC 7874, WebRTC Audio Codec and Processing Requirements. rfc
  • RFC 6455, The WebSocket Protocol. rfc
  • IETF draft draft-ietf-rmcat-gcc-02, A Google Congestion Control Algorithm for Real-Time Communication. draft
  • W3C, WebRTC: Real-Time Communication in Browsers. spec

Similar systems

Share the most building blocks with this one.

ucaas 59 %voip 47 %telehealth 47 %remote-desktop 45 %group-voice 40 %