← Back
Home › Systems › Simulation and scientific computing

nowcasting

radar precipitation nowcasting

DeepMind DGMR, RainViewer

Open the nowcasting pagenowcasting DAG workflow executiondistributed: Run a graph of dependent tasks by releasing each one when its inputs are done, longest chain first, and skip any task whose input hashes are unchanged.›Partitioned commit logdistributed: A topic split into replicated append-only partitions: key hash keeps per-key order, offsets give position, and a high watermark limits readers to fully replicated records.›Map projectionsgeometry: Web Mercator, UTM and equal-area projections, datums and coordinate transformsMap tile pyramidgeometry: Z/x/y tile pyramids, quadkeys and overviews for multi-resolution maps and imageryImage warping remapgraphics: Resampling with lookup maps: resize, lens undistortion, homographies, bilinear/bicubicData loading pipelinesmachine-learning: Turn files on disk into shuffled, sharded, resumable GPU batches fast enough that the accelerator never waits.›Evaluation metricsmachine-learning: Precision, recall, ROC AUC, average precision, log loss, NDCG and pass@k computed correctly, plus paired confidence intervals for comparing models.›Recurrent networksmachine-learning: GRU and LSTM cells for streaming sequence modelsMotion estimationmedia: Block matching and sub-pixel motion search for inter predictionConvolutional networksml-training: Layers that slide a small shared kernel over a grid, giving translation equivariance and few parameters; still the most compute-efficient choice for many vision tasks.›Diffusion modelsml-training: Train a network to undo Gaussian noise at every noise level, then generate by integrating from pure noise to data; the standard for images, video and audio.›Mixed-precision trainingml-training: Run matmuls in bf16/fp16/fp8 on tensor cores while keeping master weights, optimizer state and reductions in fp32; about 2x speed, half the activation memory.›Common alerting protocolnetworking: CAP alert message format: areas as polygons/circles, severity, urgency, multilingual info blocksFast Fourier transformnumerics: Computes the DFT in O(n log n) by recursively splitting into even and odd halves; the basis of fast convolution and spectral methods.›Semi lagrangian advectionphysics: Trace characteristics back and interpolate; unconditionally stable large time stepsChunked array storagestorage: Cut an N-dimensional array into a regular grid of independently compressed blocks, so a slice read touches only the overlapping blocks and writers to different blocks run in parallel.›Durable job queuedistributed: Persisted tasks that workers lease for a visibility timeout, ack on success and retry with backoff; at-least-once execution that survives worker and queue crashes.›Topological sortgraphs: Order the vertices of a DAG so every edge points forward; the core of build systems, schedulers and dependency resolution, with cycle detection for free.›Content-addressed storagestorage: Store each blob under the hash of its bytes: immutable, deduplicated by construction, self-verifying on every read, and composable into Merkle trees.›Exponential backoff with jittersystems: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.›Fast hash functionsdata-structures: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.›Consumer offsetsdistributed: Each consumer group stores one integer per partition, the next position to read, so a restarted consumer resumes where it left off with at-least-once delivery.›Primary-backup replicationdistributed: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›Segmented logdistributed: Split an append-only log into files named by their first offset, write only the newest, and find any record by binary search plus a sparse index.›Zero-copy I/Onetworking: Move bytes between files, sockets and NICs by passing page references instead of copying through user space: sendfile, splice, MSG_ZEROCOPY, kTLS.›mmap and the page cachehardware-os: How read() and mmap reach file data through the kernel page cache, what readahead and madvise do, and why databases use their own buffer pool instead.›Sequence packingml-training: Packing variable-length sequences into fixed-size batches with attention masks to avoid padding wastePRNGs (PCG, xoshiro)numerics: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.›+ 16 moreDead-letter queuedistributed: A side queue for messages that keep failing: after N delivery attempts or a permanent error, park them with the reason so the main queue keeps moving.›Idempotency keysfinance: Client-chosen unique key per logical operation, stored with a request fingerprint and the final response, so retries replay the first result instead of acting twice.›BFS and DFSgraphs: The two basic graph traversals; BFS gives hop distances and levels, DFS gives discovery/finish order for cycles, components and topological order.›Merkle dagdata-structures: Content-addressed DAG of hashed nodes (git trees/commits) for dedupe and verifiable historyAtomic file replacementhardware-os: Write temp file, fsync, rename over target, fsync directory: crash-safe whole-file updatesCryptographic hashes (SHA-2,BLAKE3)security: Collision-resistant fingerprints of data; SHA-256 for interoperability, BLAKE3 for speed, HMAC/HKDF for keyed uses.›Leasesdistributed: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.›Write-ahead logstorage: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›Fencing tokensdistributed: A number that grows with every lock grant and is checked by the protected resource, so a paused ex-holder's late write is rejected.›Message sequence numbersdistributed: A per-stream counter on every message lets the receiver detect loss, duplicates and reordering, ack cheaply and resume after reconnect with "send from N".›Health checkingsystems: Decide per backend whether to route to it using probes and real-traffic errors, with consecutive-count hysteresis, capped ejection and a panic fallback.›Virtual memory and the TLBhardware-os: How page tables and TLBs translate every address, what TLB reach means, and when 2 MiB or 1 GiB huge pages pay off on Linux.›+ 3 moreSparse matrices (CSR)numerics: Compressed sparse row stores only nonzeros as row pointers, column indices and values; SpMV streams them once, so it is memory-bandwidth bound.›CRC32C checksumshardware-os: The Castagnoli CRC-32 with dedicated instructions on x86 (SSE4.2) and ARMv8. It is the standard fast integrity check for storage blocks, logs and network frames.›fsync, O_DSYNC and O_DIRECThardware-os: What fsync, fdatasync, O_DSYNC, O_DIRECT and F_FULLFSYNC really guarantee on Linux and macOS, and the exact call sequences for crash-safe files.›

Similar systems

Share the most building blocks with this one.

flood-forecasting 24 %earth-observation 21 %ml-weather-model 16 %