online-judge

How it is built

Architecture overview

An online judge accepts source code from strangers, runs it against hidden tests, and answers with a verdict (accepted, wrong answer, time limit, runtime error, compile error) and, in a contest, a place on a live scoreboard. Three things are hard. The code is hostile or at least broken, so every run happens in a sandbox that has no network, a minimal file system and hard limits on time, memory, processes and output. The result must be fair, so the same program must get the same time on a Monday morning and in the final minute of a contest, which constrains how judge machines are set up. Load arrives in bursts, mostly in the last minutes of a contest, so judging is a queue with workers, not a request-reply call. The one idea that makes it work: separate the untrusted, powerless part (the sandboxed run) from the trusted, privileged part (queue, checker, scoring) and let only a narrow, well-defined result cross the boundary: an exit status, a few numbers and an output file.

Parts:

• Web and API — accept submissions, show verdicts, enforce submission rate — Token bucket, Distributed rate limiting
• Storage — sources, test data, outputs, compiled artefacts — Object storage, Content-addressed storage, Signed URLs
• Judge queue — hand each submission to exactly one live judge, recover if it dies — Durable job queue, Leases, Fencing tokens, Dead-letter queue
• Sandbox — run one untrusted program with limits and no reach — Process sandboxing, Cgroups and namespaces, Least privilege and isolation, seccomp-bpf (no page yet), cpu-isolation-and-pinning (no page yet)
• Checker and scoring — compare output, apply the contest's scoring rule — output-checkers (no page yet), contest-scoring-rules (no page yet)
• Scoreboard and push — ranked standings, live updates — Skip list, Pub/sub fan-out, WebSockets
• Ratings — long-term skill estimate from contest results — Elo rating, Glicko-2
• Plagiarism check — find copied solutions after the contest — winnowing-fingerprints (no page yet), Near-duplicate detection with MinHash
• Operations — keeps the judging service running, recoverable and measurable — Process supervision, Graceful draining, Mutual TLS, Point-in-time recovery, Primary-backup replication, hdr-histogram (no page yet), Deterministic simulation testing, Coverage-guided fuzzing

 contestant ──submit (rate-limited)──► web/API ──► DB: submission row (state=queued)
                                          │ source ──► object store
                                          ▼
                                    judge queue (lease per attempt)
                                          │ pull
                        ┌─────────────────▼────────────────────────────┐
                        │ judge host (dedicated cores, no other load)  │
                        │  fetch source + tests (cache by hash)        │
                        │  compile ─► sandbox ─┐                        │
                        │  for each test:      │ status, time, memory   │
                        │   run ─► sandbox ────┴─► output file          │
                        │   checker (trusted) ─► OK / WA                │
                        └─────────────────┬────────────────────────────┘
                                          ▼ verdict (+ attempt id)
              DB: judging row ─► scoreboard cache ─► pub/sub ─► WebSocket clients
                                          └──► after contest: ratings, plagiarism jobs

How the parts fit together

Sandbox

The sandbox is the part that can go wrong in the worst way, so it is built in layers, each of which assumes the layer above it failed (Least privilege and isolation). The documented building blocks, taken from the manual of isolate, a sandbox built for programming contests and used by CMS, and from the kernel documentation:

• A separate, unprivileged identity. isolate runs setuid to root only to set things up; the program inside runs as a non-privileged user ID, a different one for each box id so that parallel boxes do not share an identity.
• A private view of the system (Cgroups and namespaces). The program gets its own file system view containing only directories requested by rules: by default /bin, /lib, /usr read-only, a writable /box and /tmp, and a /proc that shows only its own user's processes. It gets a new network namespace with only a loopback device, so a program may create a socket but has nowhere to connect it. Bound directories are mounted read-only without devices or setuid binaries unless a rule says otherwise.
• Resource limits. CPU time (--time, with a larger wall-clock limit as a guard against programs that sleep), address space, stack, open files, size of each created file, core files (default zero) and the number of processes (default: one). With one process, limits are per process; for programs that start several processes or threads, isolate uses cgroups (v2) to limit and report memory of the whole group, and reports whether the out-of-memory killer fired. Swap must be off, since memory limits do not cover swapped-out data (isolate manual).
• System-call filtering (seccomp-bpf, no page yet). A seccomp filter is a small BPF program, run by the kernel on each system call's number and arguments, that can allow, fail or kill (kernel doc "Seccomp BPF"). isolate itself uses it narrowly: it does not try to enumerate all safe calls, because namespaces already limit which objects can be reached. It forbids only a few calls that the kernel does not namespace and that could carry data between two boxes on the same machine: keyrings, VM sockets, file locks on shared inodes, io_uring and legacy architectures. This is a useful design rule: filter what namespaces cannot contain, instead of writing a long allow-list that breaks compilers and runtimes.
• Different systems, same layers. DMOJ's judge runs on Linux and FreeBSD, grades with ptrace and seccomp, and does not need a root user on Linux (judge-server README). DOMjudge's runguard wrapper runs compilers and submissions in a prebuilt chroot tree used read-only, as an unprivileged user (DOMjudge manual).

Contract with the rest of the system: the sandbox takes (command, limits, input file, box directory) and returns (status, exit code or signal, CPU time, wall time, peak memory, output file). isolate's meta file reports exactly such a record, with two-letter status codes: RE (non-zero exit), SG (died on a signal), TO (timed out) and XX (internal error of the sandbox). Never map XX to a contestant verdict.

One note from the isolate manual that decides architecture: running isolate in containers is not recommended, because container managers usually do not delegate cgroups properly and other workloads disturb the measured time.

Judge host and timing fairness

Time limits are only fair if the same program gets about the same time every run. The isolate manual lists what to change on a judge machine: disable address-space randomization, set the CPU frequency governor to "performance", consider disabling turbo boost, run evaluations on a single CPU core (the scheduler's migrations cost cache warmth), pin to a homogeneous subset on hybrid P/E-core CPUs, disable transparent huge pages and swapping. DOMjudge's manual says the same in operational form: bind each judge daemon to one CPU core, avoid cores that share a hyperthread with the system's CPU 0, and, as a rule of thumb, plan one judgehost per 20 teams. cpu-isolation-and-pinning (no page yet) is the block that explains the mechanism. The consequence: a judge host does one run at a time per pinned core and is not shared with other work, so throughput scales with dedicated cores, not with clever packing.

Queue and recovery

A judging job is long (compile plus many tests; DOMjudge says one solution can tie up a judgehost for several minutes depending on time limits and test count) and judges are machines that reboot. So the queue is a Durable job queue where a judge takes a job under a Lease: if the judge does not renew or finish in time, the lease expires and another judge takes the job. Two judges can then hold the same job for a moment (the first was only slow), so the result carries an attempt number and the database accepts only the newest one, which is the job of Fencing tokens. A job that crashes judges repeatedly ends in a Dead-letter queue with a human-visible reason, not in an endless retry loop.

Fairness is a queue policy. DOMjudge gives a submission from a team that has no earlier submission waiting a higher priority than one from a team that already has submissions in the queue, so one team spamming submissions cannot starve the others (DOMjudge manual, "Flow of a submission"). Rejudging (after a bad test case or compiler) re-enters the flow with the original submit time, and the old judging is kept but marked invalid; a "rejudging set" lets the jury inspect all new results before making them effective.

Storage

Sources, test data and outputs are blobs. Object storage holds them; judges download through Signed URLs so they need no database credentials. Test data is large and reused by thousands of runs, so judges cache it by content hash (Content-addressed storage): a changed test file gets a new hash, and a stale cache cannot silently serve old tests. Output from the sandbox is capped (--fsize) so a program printing in a loop cannot fill the disk.

Checker and scoring

The checker is trusted code written by the problem setter; the contestant's output is untrusted data to it. It runs outside the sandbox's contestant identity and compares the output with the expected answer, either by exact or token comparison or by a custom validator (needed whenever many answers are right). DMOJ lists support for interactive tasks, signature-graded tasks, runtime data generators and custom output validators (README).

Scoring turns verdicts into points; DOMjudge's manual documents the two main rules. ICPC style (pass-fail): rank by the number of problems solved, then by total penalty time, then by the time of the last accepted submission. IOI style (partial scoring): test cases are grouped; each group has an accept score and an aggregation (sum, min, max or average) and may continue testing after a failure; the team keeps its best score. The first can stop at the first failing test; the second must keep going, so make it a property of the problem.

Scoreboard and push

The scoreboard changes on every verdict, so recomputing it from all submissions is the wrong default. DOMjudge keeps a scorecache table (one row per team and problem) and a rankcache table (one row per team) and updates both when a judging completes. Where a service serves the standings from memory, an ordered structure answers "what is rank r" and "update this team's score" in O(log n); Redis implements its sorted sets with a Skip list plus a hash table (Redis documentation). A scoreboard freeze near the end of a contest just stops updating the public view while the real one keeps changing.

To show changes live, each update is published once and fanned out (Pub/sub fan-out) to browsers over WebSockets: one encoded message shared by all subscribers, a bounded queue per connection, and a slow client gets the latest standings instead of a backlog. CMS goes one step further and separates its RankingWebServer on purpose, so mirrors can serve the public while the services connected to the database stay hidden (CMS documentation).

Submission rate and abuse

A Token bucket per user limits submissions: an average rate plus a small burst; with several web servers see Distributed rate limiting. The public Moss service shows why: after robots submitted tens of thousands of jobs per day it began enforcing 100 submissions per user per day (Moss page, 13 Nov 2022).

Ratings and plagiarism

After a contest, ranks are turned into rating changes with an Elo-family update (Elo rating; Glicko-2 adds an uncertainty per player, which suits people who compete rarely). A contest has many participants at once, so one would compare each participant's rank with the rank expected from the others' ratings (reasoning; the block page gives the pairwise rule).

For copying, winnowing-fingerprints (no page yet) is the block behind Moss: hash every k-gram of a normalised program, slide a window over the hashes and keep the smallest in each window; the paper shows that any shared passage above a length threshold is guaranteed to yield a common fingerprint, and that matches below the noise threshold are ignored (Schleimer, Wilkerson and Aiken, SIGMOD 2003). For large corpora, MinHash with LSH finds candidate pairs first.

Operations

Process supervision keeps the judge daemon, the web workers and the push gateway alive: a parent starts each, reaps its exit and restarts it under a delay and burst limit, so a judge that crashes on one bad submission comes back without re-running the crash in a tight loop. A judge host is stopped with SIGTERM first and SIGKILL only after a deadline. Graceful draining is what a judge does on that SIGTERM: stop pulling from the queue, finish the test run in hand (or give the attempt back so its lease expires and another judge takes it), and leave the host free for a deploy in the middle of a contest. WebSocket gateways drain the same way and tell clients to reconnect elsewhere.

Mutual TLS gives every internal hop a verified machine identity: a judge proves its certificate to the queue and the object store before it may fetch tests or report a verdict, so a stray machine cannot post verdicts, and certificates rotate without a redeploy. The sandboxed program never sees these credentials; only the trusted judge process holds them.

Primary-backup replication and Point-in-time recovery protect the submission and judging database. The primary orders writes and ships them to a standby, so a lost database host costs a failover, not the contest. Base backups plus the archived change log let an operator rebuild the state as of an instant, for example just before a bad rejudge script rewrote verdicts, and branch from there.

hdr-histogram (no page yet) records the latency that contestants feel, from submit to verdict, as percentiles with bounded relative error; the tail in the last minutes of a contest is the number that shows queue depth turning into waiting. Deterministic simulation testing runs the queue, leases and fencing tokens under a seeded scheduler with simulated crashes and clock skew, so a double-judged or lost submission replays from its seed. Coverage-guided fuzzing targets the parts that parse hostile input: the submission upload, the compiler wrapper's argument handling and the checker's reading of contestant output.

Data and control flow

One submission:

1. The web server checks the token bucket, authenticates, stores the source in object storage and inserts a submission row (state queued). Time here: milliseconds (order of).
2. A judge leases the job, downloads source and tests (cache hits skip the tests), and compiles inside a sandbox with its own limits. Failure ends in "compile error" with the compiler output.
3. For each test: a fresh box is initialised, input copied in, the program run with limits, the output file fetched, the box removed. Most of the wall time goes to tests (many runs, each up to the time limit) and to compiling slow languages, not to the queue.
4. The checker reads output and expected answer and returns OK or WA (or a score). The first failing test ends a pass-fail judging.
5. The judge posts the verdict with its attempt number. The database accepts it only if the attempt is current.
6. The scoreboard cells for (team, problem) update; the new standings are published; clients redraw.

A rejudge: an operator picks a problem and a reason; affected submissions re-enter the queue, are judged into a rejudging set, and the jury applies or cancels the set after comparing old and new results (DOMjudge manual).

Scaling

In the order you would pull the levers:

1. Judge throughput is cores times test time. Add judge hosts (DOMjudge: one per 20 teams as a rule of thumb), stop at the first failing test when the scoring rule allows, cache compiled binaries and test data.
2. Timing noise limits how many runs you can pack. Dedicate and pin cores; do not share with other work; run outside containers (isolate manual).
3. Burst at the end of a contest. Queue depth, not latency of one run, becomes the user's wait. Prioritise by fairness, keep rejudges low priority, and over-provision judges for the contest window (they can be shut off afterwards).
4. The database holds submissions, judgings and scoreboard caches; keep the hot scoreboard in a cache table or memory and the full history append-only. CMS keeps the whole contest state in PostgreSQL and lets all other services be stopped and restarted independently (CMS documentation).
5. Push fan-out grows with spectators. Serve the public standings from a read-only mirror, coalesce updates and bound each connection's queue.

Failure modes

Real implementations

For Codeforces and LeetCode, no open-source code or engineering document was found for this page, so nothing is claimed about how they are built.

Sources

• M. Mares, B. Blackham, A New Contest Sandbox, Olympiads in Informatics 6 (2012), the design paper of isolate. journal · author copy
• M. Mares, Security of Grading Systems, Olympiads in Informatics 15 (2021), DOI 10.15388/ioi.2021.04. author copy
• isolate manual page (limits, namespaces, cgroups, meta-file, reproducibility, system-call restrictions). man page
• Linux kernel documentation, Seccomp BPF (SECure COMPuting with filters). doc · Control Group v2: doc · namespaces(7): man
• DOMjudge manual: overview, judgehost installation, judging topics, scoring.
• CMS documentation, Introduction (services, PostgreSQL, security considerations). docs
• DMOJ judge-server README. repo
• S. Schleimer, D. Wilkerson, A. Aiken, Winnowing: Local Algorithms for Document Fingerprinting, SIGMOD 2003. paper · Moss
• Redis documentation, Sorted sets (skip list plus hash table). docs