short-video

How it is built

Architecture overview

A short-video platform lets creators upload clips of seconds to a few minutes and gives every viewer an endless, personalised vertical feed where the next clip starts the instant they swipe. Three loops make it work. An upload loop turns each file into a ladder of small, cacheable renditions. A serving loop picks a few dozen clips for one viewer out of millions (retrieve, rank, filter) and hides the network behind a client that already holds the first seconds of the clips that might come next. A learning loop logs what the viewer did with each clip (finished, replayed, skipped, liked) and feeds it back into the models within minutes. The one idea that holds it together: the feed is a funnel of cheap-to-rich models fed by a fast feedback stream, while the media bytes are static files a CDN can serve without knowing who is watching.

Parts:

• Upload and ingest — accepts, checks, names and queues the file — Resumable uploads, Durable job queue, Distributed ID generation, Object storage, CRC32C checksums
• Transcode pipeline — encodes a ladder of renditions in parallel chunks — Video transcoding ladder, chunked-parallel-encoding (no page yet), video-codecs (no page yet), GOP structure, Adaptive quantization, Perceptual quality metrics, Image resampling filters
• Delivery — manifests and segments cached near the viewer — CDN and edge, HTTP caching semantics, HTTP range requests, S3-FIFO, TLS 1.3, adaptive-streaming-manifests (no page yet), media-containers (no page yet)
• Client playback and prefetch — picks renditions, holds the next clips' first chunks — Adaptive bitrate streaming, Exponential smoothing (ETS, Holt-Winters)
• Event log — one durable stream of impressions and actions that everything else reads — Partitioned commit log, Consumer offsets, Windowed stream aggregation, watermarks (no page yet)
• Candidate retrieval — user vector in, a few hundred clips out — Two-tower retrieval, Embeddings, Negative sampling, Contrastive learning, HNSW
• Ranking — scores each candidate for several feedback signals — Deep ranking models for recommendation, Feature hashing (the hashing trick), Mixture of experts (MoE), embedding-table-sharding (no page yet)
• Online learning — updates embeddings and ranker from the live stream — Cuckoo hashing, Count-min sketch, Point-in-time recovery
• Exploration and experiments — gives new clips a first audience, tests every change — Multi-armed bandits, Bayesian rate estimation, A/B testing (online controlled experiments), Deterministic bucketing, Experiment layering (overlapping experiments)
• Integrity — spots re-uploads and near-duplicates — Perceptual hashing
• Operations and verification — keeps the upload, feed and event services running, authenticated, measured and tested — Process supervision, Graceful draining, Mutual TLS, Primary-backup replication, hdr-histogram (no page yet), Coverage-guided fuzzing, Deterministic simulation testing

 creator ─► upload ─► object store ─► job queue ─► transcode ─► renditions + manifests ─► origin ─► CDN ─► phone
                │                                      │                                                   ▲
                ▼                                      ▼ item vector                      first chunks of  │
          clip id + metadata ─────────────────► candidate index (ANN)                      next clips      │
 viewer ─► feed request ─► user vector ─► retrieve ─► rank (multi-task) ─► filter ─► manifest of clip ids ───┘
                                              ▲            ▲
                                              │ new weights│ new embedding rows (minutes)
 phone ─► impressions + actions ─► event log ─► joiner ─► training examples ─► trainer

How the parts fit together

Upload and ingest

Phones upload over bad links, so the endpoint speaks a resumable protocol (Resumable uploads) and checks each part (CRC32C checksums). The file lands in Object storage under an id from a time-ordered scheme (Distributed ID generation) that needs no coordination between ingest nodes. A Durable job queue hands the clip to transcoding; the queue, not the uploader, owns retries, because a transcode can fail halfway and must be repeatable. Contract: accepted means bytes and an ingest job are durable; published means renditions exist and checks passed. One check is Perceptual hashing, which compares frames or audio fingerprints against known re-uploads (moderation is a system of its own).

Transcode pipeline

Clips are short, so the unit of parallelism is the chunk: split at keyframes, encode chunks on many workers, stitch (chunked-parallel-encoding, no page yet). The Video transcoding ladder picks the renditions from measured rate-quality points scored with Perceptual quality metrics; GOP structure decides whether keyframes line up across renditions, which makes clean switching possible. Adaptive quantization spends bits where artefacts show, and Image resampling filters make thumbnails and downscaled renditions.

The costly decision is how much encoding a clip deserves. Meta describes it for Reels: a clip first gets cheap hardware AVC encodes whose quality and bitrate are recorded; only after its watch time passes a threshold does software encoding run at the points chosen on the convex hull of (resolution, quality) pairs, with AV1 added as a second-stage codec (Meta, 2023). Instagram's related post reports a 94 percent cut in the compute for basic adaptive encodings by repackaging the progressive encodings instead of transcoding the upload again (Meta, 2022). The lesson: most clips are watched a few times, so use a cheap ladder at upload and an expensive one for clips the recommender has proven.

Delivery

Renditions are cut into segments with manifests (HLS playlists: RFC 8216; adaptive-streaming-manifests and media-containers, no page yet), so every request is a plain GET of an immutable file, often a byte range (HTTP range requests). That is why the CDN and edge works: freshness follows HTTP caching semantics, the cache needs an eviction rule that survives a long tail of one-off clips (S3-FIFO is built for that), and transport is TLS 1.3. New renditions get new names, so cached copies are never edited in place.

Client playback and prefetch

The player's rendition choice is Adaptive bitrate streaming with a throughput estimate such as Exponential smoothing (ETS, Holt-Winters). Short video changes one thing: the viewer's thumb decides playback order. The Dashlet study (NSDI 2023) measured TikTok in the wild and reports: size-based chunks (the first chunk is the first megabyte, or the whole file if smaller), the first chunk of the next five videos always pre-buffered whatever the network, and a manifest for an ordered group of 10 videos with one logical buffer each. Dashlet replaces that fixed rule with a swipe-statistics model that picks chunks out of order and reports 28 to 101 percent better QoE than TikTok and 30 percent fewer wasted bytes (Li et al., 2023). Read the TikTok behaviour as a 2022-23 snapshot. The contract with the feed: it returns a batch of upcoming ids early, so the client can prefetch.

Event log

Every impression, watch with duration, replay, like and skip is an event on the Partitioned commit log, keyed so one user's events stay ordered and replayable. Each consumer (trainer, metrics, back-filler) tracks its own position with Consumer offsets. Windowed stream aggregation (with watermarks, no page yet) turns events into features such as a clip's completion rate over the last hour. Monolith's streaming engine has this shape: one Kafka queue for user actions, one for features, a Flink job that joins them into training examples on a third queue, read by both online and batch training (Liu et al., 2022).

Candidate retrieval

Retrieval must be sublinear in the corpus. Two-tower retrieval learns a user tower and an item tower into one space; item vectors are computed offline and indexed, and a request costs one user-tower pass plus a nearest-neighbour search. YouTube's 2016 system poses candidate generation as extreme multiclass classification over millions of videos, trained with sampled negatives (several thousand per example, over 100 times faster than a full softmax) and served as a dot-product nearest-neighbour lookup (Covington et al., 2016). They add that A/B results were not particularly sensitive to the nearest-neighbour algorithm, so choose the index on operations: HNSW accepts insertions as clips arrive; IVF-PQ compresses vectors to fit more per machine.

Training uses Embeddings for ids, Negative sampling so the loss avoids every clip, and Contrastive learning when true (user, clip) pairs must outscore in-batch others. Two details from the YouTube paper carry over: feed the example's age as a feature (zero at serving) so the model does not replay last month's hits, and predict the next watch from earlier history, not a random held-out one, which leaks the future. Matrix factorization for collaborative filtering is the depth-zero case of the same model; the paper calls the neural version a non-linear generalisation.

Ranking

A few hundred candidates can afford a heavy model. A Deep ranking model maps sparse ids to embeddings, crosses them in an MLP and predicts several feedback signals per clip; a weighted sum is the score. YouTube optimised expected watch time with weighted logistic regression (positives weighted by watch time) because ranking by click promotes clips users do not finish (Covington et al., 2016). TikTok lists the signal kinds: likes, shares, follows and comments; captions, sounds and hashtags; device and account settings at lower weight; finishing a longer video counts more than weak signals, and "not interested" counts against a clip (TikTok newsroom). It also says feeds avoid two videos in a row with the same sound or creator and exclude duplicates and seen clips: filters after the model. Mixture of experts (MoE) lets conflicting tasks use different experts; a per-user Bloom filter is one cheap way to implement the "seen" check.

The id space is the memory problem. Feature hashing (the hashing trick) bounds the table but makes popular and rare ids collide; sharding the table across servers (embedding-table-sharding, no page yet) makes each lookup an all-to-all exchange.

Online learning

Interest drifts in minutes; a nightly-trained ranker is a day stale. Monolith (ByteDance, ORSUM at RecSys 2022) is the published design:

• Collisionless table. A key-value table on Cuckoo hashing gives each id its own row with worst-case O(1) lookup. Ids are admitted after a tunable occurrence threshold plus a probabilistic filter (a Count-min sketch fits, but the paper does not say), and expire when inactive. On MovieLens, hashing collided 7.73 percent of user ids and 2.86 percent of movie ids, and the collisionless table gave higher AUC.
• Two parameter servers. Training servers push to serving servers: only rows touched since the last push, at minute-level intervals; dense layers move slowly and sync about daily. Few rows change in a short window, so the diff is small and serving needs no second copy of a multi-terabyte model.
• Cheap fault tolerance. Training servers are snapshotted daily (Point-in-time recovery is the general idea). With a very low machine failure rate, losing a day on one server in a thousand costs a negligible user fraction, so freshness wins over reliability.
• Evidence. In a Criteo simulation, average online AUC was 79.66, 79.78 and 79.80 percent at 5 hour, 1 hour and 30 minute sync intervals; a live ads A/B test favoured online over batch training on all seven days reported.

A click may arrive long after its impression, so unmatched features wait in an in-memory cache backed by an on-disk store. Negatives are sampled, and a log-odds correction at serving restores the true distribution (Liu et al., 2022).

Exploration and experiments

A new clip has no history, so the ranker has no reason to show it. YouTube trains on all watches, not only those from its own recommendations, so discovery elsewhere propagates (Covington et al., 2016). At serving time, Multi-armed bandits give a clip a small trial audience and widen it as evidence arrives, and Bayesian rate estimation turns "40 of 60 finished it" into a rate with an honest interval. The bandit framing is this page's design choice; TikTok says only that it sometimes shows videos outside a viewer's stated interests or with fewer likes.

Every model, index, ladder and prefetch rule goes through A/B testing (online controlled experiments): Deterministic bucketing assigns users by hash, and Experiment layering (overlapping experiments) runs many tests on disjoint parameters. YouTube notes offline metrics do not always agree with live A/B results.

Operations and verification

Process supervision starts and restarts the ingest gateway, the encoder workers and the feed service with a delay and burst limit, so one corrupt upload that crashes an encoder cannot cause a restart storm. Graceful draining lets a feed server finish the requests in flight and an encoder finish its chunk before it leaves; unfinished chunks go back to the job queue, and the client retries the feed call against another server. Mutual TLS authenticates the calls between ingest, encoders, the origin store, the event log and the ranker, so only the pipeline can publish a rendition and only the event gateway can append impressions. Primary-backup replication protects the small authoritative stores (clip metadata, creator and policy state): one primary orders writes and a standby takes over, because a clip without metadata cannot be served. hdr-histogram (no page yet) records latency percentiles of the feed request (retrieval plus ranking) and of time to first frame, since a swipe feels slow at the tail rather than at the mean; the ranker's latency budget is checked against that percentile. Coverage-guided fuzzing mutates uploaded containers, subtitles and metadata fields against the probe and demuxer, which parse untrusted files; inputs that reach new code are kept. Deterministic simulation testing replays the upload-to-publish workflow and the event-log consumers under a seeded clock with delayed, duplicated and lost messages, so a clip published twice or an impression counted twice replays from its seed.

Data and control flow

1. A clip, from upload to first view.

1. The phone uploads resumable parts with checksums; the id is minted and an ingest job queued.
2. The pipeline splits at keyframes and encodes chunks in parallel into the cheap ladder; thumbnails and manifest are written; duplicate and safety checks run alongside.
3. The clip is published: metadata, embedding and initial features enter the candidate index and the feature store (see the feature-store system); the CDN pulls segments from the origin on first request.
4. A trial audience sees it; impressions and watch events flow into the log.
5. Past a watch-time threshold the second encoding stage runs and writes better renditions under new names.

2. One feed request.

1. The server loads the user's features and runs the user tower.
2. Retrieval returns a few hundred nearest clips, possibly merged with other sources (follows, trending).
3. The ranker scores them; the funnel goes from millions to hundreds to dozens (Covington et al., 2016).
4. Filters drop seen, duplicate and blocked clips; diversity rules reorder.
5. The server returns an ordered manifest of ids and rendition URLs; the client prefetches the next clips' first chunks. Time goes to feature lookup, search, ranking and the first byte from the edge.

3. One feedback cycle.

1. The player logs an impression and watch events with a request id.
2. The joiner pairs the impression's features with action labels, waiting for late actions, and writes examples.
3. The online trainer updates sparse rows and dense layers.
4. Every minute or so the touched rows are pushed to serving; the user's next request sees them.

Scaling

In the order the limits usually bite:

1. Egress and CDN cost. Bytes served dominate. Levers: better codecs and ladders (extra encoding pays back at high view counts, hence two stages), cache hit rate, prefetch that wastes fewer bytes.
2. Embedding memory. Ids grow with users and clips. Levers: admission and expiry, sharding, smaller or quantised rows.
3. Index freshness against recall. New clips must be searchable in minutes. Levers: an index that accepts inserts, a small fresh-clip index merged into the main one, periodic rebuilds.
4. Ranking latency. A fixed budget caps model size. Levers: fewer candidates, caching user-side work, smaller models.
5. Feedback delay. Levers: more log partitions, a scaled joiner, pushing only touched rows, looser snapshots.

Failure modes

Real implementations

Sources

• Z. Liu et al., Monolith: Real Time Recommendation System With Collisionless Embedding Table, ORSUM at RecSys 2022. paper
• P. Covington, J. Adams, E. Sargin, Deep Neural Networks for YouTube Recommendations, RecSys 2016. page; DOI 10.1145/2959100.2959190
• Z. Li, Y. Xie, R. Netravali, K. Jamieson, Dashlet: Taming Swipe Uncertainty for Robust Short Video Streaming, NSDI 2023. paper
• Meta Engineering, Reducing Instagram's basic video compute time by 94 percent, 4 Nov 2022. post
• Meta Engineering, How Meta brought AV1 to Reels, 21 Feb 2023. post
• TikTok Newsroom, How TikTok recommends videos #ForYou. post
• R. Pantos (ed.), W. May, HTTP Live Streaming, RFC 8216, 2017. RFC