Object storage

Deep dive

Storage classes (AWS names; others similar)

Decision table

Code

Fast parallel upload with the AWS CLI (works against R2/B2 with --endpoint-url):

aws configure set default.s3.max_concurrent_requests 64
aws configure set default.s3.multipart_chunksize 64MB
aws s3 cp ./dump.tar.zst s3://backups/db/2026-10-20.tar.zst \
  --endpoint-url https://<account>.r2.cloudflarestorage.com

Lifecycle rules: expire old versions and incomplete multipart uploads (the forgotten cost).

{ "Rules": [
  { "ID": "abort-mpu", "Status": "Enabled", "Filter": {},
    "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 3 } },
  { "ID": "tier-logs", "Status": "Enabled", "Filter": { "Prefix": "logs/" },
    "Transitions": [ { "Days": 30, "StorageClass": "STANDARD_IA" },
                     { "Days": 180, "StorageClass": "GLACIER_IR" } ],
    "Expiration": { "Days": 730 } },
  { "ID": "old-versions", "Status": "Enabled", "Filter": {},
    "NoncurrentVersionExpiration": { "NoncurrentDays": 30 } } ] }

Presigned URL so clients upload directly, bypassing your servers (Rust, aws-sdk-s3):

use aws_sdk_s3::presigning::PresigningConfig;
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let conf = aws_config::load_defaults(aws_config::BehaviorVersion::latest()).await;
    let s3 = aws_sdk_s3::Client::new(&conf);
    let req = s3.put_object()
        .bucket("uploads").key("u/123/avatar.webp").content_type("image/webp")
        .presigned(PresigningConfig::expires_in(Duration::from_secs(300))?)
        .await?;
    println!("{}", req.uri());                       // client PUTs to this URL within 5 minutes
    Ok(())
}

Efficiency

• Cost: S3 sustains at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned prefix, with no limit on prefixes (S3 performance docs, as of 2026-10). Small objects are bound by request count and per-request latency (order of 10-100 ms first byte); large objects by connection bandwidth. R2 has no egress bandwidth charge (R2 pricing, as of 2026-10). Bound: requests for small objects, egress price for large ones.
• Levers:
  • parallel connections and ranged GETs — throughput scales with parallelization (S3 performance docs) — large objects
  • retry the slowest 5 % of large requests (over 128 MB) — cuts tail latency (S3 design patterns doc) — big transfers
  • CloudFront or ElastiCache in front — fewer direct requests, "orders of magnitude" lower GET latency with ElastiCache (S3 design patterns doc) — hot objects
  • spread keys over several prefixes — request limits apply per prefix (S3 performance docs) — hot key spaces
  • pack tiny objects into larger files — fewer billed requests (this page) — logs, events
• Naive → standard → state of the art:
  • one sequential GET per small object, LIST to find keys · one request and 10-100 ms per object (order of)
  • parallel ranged GETs or multipart, manifest instead of LIST, lifecycle rules · connection count times per-connection rate
  • cache tier, packed columnar files such as Parquet, same-region or zero-egress reads · requests per byte and egress both fall

Implementation notes

• Throughput: one connection gives order of 50-100 MB/s; use 16-64 parallel ranged GETs or multipart parts (8-64 MB) to saturate 10-100 Gbit/s.
• S3 scales per prefix to order of 3,500 writes and 5,500 reads per second; spread hot keys over prefixes or let it auto-partition and back off on 503 SlowDown.
• S3 is strongly read-after-write consistent since 2020; other S3-compatible stores may differ, check before relying on list-after-write.
• Use conditional writes (If-None-Match: *, If-Match on ETag, supported on S3 since 2024 and on R2) to build atomic manifests for logs and table formats.
• Enable versioning on anything you can't regenerate; it protects against deletes and ransomware.

Pitfalls

• Egress from a hyperscaler bucket to users or another cloud: 100 TB/month ≈ $5-9k.
• Minimum storage durations and minimum object sizes in IA/Glacier classes bill tiny objects as 128 KB and early deletes as full months.
• LIST is slow and paid; keep your own index (a manifest or DB) instead of listing millions of keys.
• Public buckets leak data; block public access account-wide and serve via CDN with signed URLs or origin access control.
• Incomplete multipart uploads are invisible in the console and billed forever without a lifecycle rule.

Implementations

Sources

• AWS, Amazon S3 pricing pricing and Amazon S3 User Guide: performance guidelines docs
• Cloudflare, R2 pricing pricing; Backblaze, B2 Cloud Storage pricing pricing
• Andy Warfield et al., Building and operating a pretty big storage system called S3, All Things Distributed blog, 2023. post