Deep dive
Where money leaks (check in this order)
| Leak | Typical fix | Saving |
|---|---|---|
| Idle dev/test environments 24/7 | Schedules (stop nights/weekends), ephemeral preview envs | ~65 % of their cost |
| Oversized instances (CPU < 20 %) | Rightsize one size down, Arm instances | 20-50 % |
| Unattached volumes, old snapshots, orphan IPs/LBs | Weekly sweep, lifecycle policies | Small each, adds up |
| Object storage without lifecycle | Tiering + incomplete-multipart cleanup | 30-80 % of storage line |
| Egress to users | CDN, R2/B2, compression | Often 50-90 % of egress |
| NAT gateway processing | VPC endpoints for S3/DynamoDB/ECR, IPv6 egress-only | Order of $0.045/GB avoided |
| Cross-AZ traffic | Zone-aware routing, rack-aware Kafka consumers | Order of $0.02/GB round trip |
| Logs and metrics volume | Sampling, retention, drop debug | 30-70 % of observability |
| On-demand for the steady base | Savings Plans / CUDs on ~70-80 % of the baseline | 30-50 % |
Decision table
| Situation | Choice | Why |
|---|---|---|
| Steady baseline for 1+ year | Compute Savings Plan / CUD on the floor of usage, not the peak | Flexible across instance types; peaks stay on demand/spot |
| Stateless, retryable work (CI, batch, render) | Spot with diversified instance types | 60-90 % off; retry is cheap |
| Bandwidth-heavy product | Move the serving path to a CDN or commodity provider | Egress dominates; commitments don't discount it much |
| Many teams, unclear owners | Mandatory tags + per-team showback report | People cut what they see |
| GPU training | Spot/neocloud + checkpointing, idle shutdown | GPUs idle at $2-8/h each |
| Bill > ~$50k/month | Negotiate private pricing / EDP, and price a hybrid move | Discounts are negotiable at scale |
Code
Budget alert with Terraform (AWS):
resource "aws_budgets_budget" "monthly" {
name = "total-monthly"
budget_type = "COST"
limit_amount = "5000"
limit_unit = "USD"
time_unit = "MONTHLY"
notification {
comparison_operator = "GREATER_THAN"
threshold = 80
threshold_type = "PERCENTAGE"
notification_type = "FORECASTED"
subscriber_email_addresses = ["ops@example.com"]
}
}
resource "aws_ce_anomaly_monitor" "svc" {
name = "per-service"
monitor_type = "DIMENSIONAL"
monitor_dimension = "SERVICE"
}
Find the top cost drivers by usage type (where egress, NAT and cross-AZ hide):
aws ce get-cost-and-usage --time-period Start=2026-09-01,End=2026-10-01 \
--granularity MONTHLY --metrics UnblendedCost \
--group-by Type=DIMENSION,Key=USAGE_TYPE \
--query 'ResultsByTime[0].Groups[].[Keys[0],Metrics.UnblendedCost.Amount]' --output text \
| sort -k2 -nr | head -25
# look for: DataTransfer-Out-Bytes, NatGateway-Bytes, DataTransfer-Regional-Bytes (cross-AZ), EBS:SnapshotUsage
Sweep unattached volumes:
aws ec2 describe-volumes --filters Name=status,Values=available \
--query 'Volumes[].[VolumeId,Size,CreateTime]' --output table
Efficiency
- Cost: the meters are compute-hours, GB stored, GB sent out, requests, and NAT and cross-zone GB. Savings Plans give up to 72 % off covered compute (AWS Savings Plans User Guide); Spot offers up to 90 % off with a two-minute interruption notice (AWS Spot page, EC2 User Guide); gp3 volumes cost 20 % less per GiB than gp2 (EBS docs). All as of 2026-10. Bound: the largest usage type, found by sorting.
- Levers:
- Savings Plan on the usage floor — up to 72 % off covered compute (AWS) — steady baseline
- Spot with diversified instance types — up to 90 % off (AWS) — retryable work
- gp2 to gp3 — 20 % lower price per GiB, 3,000 IOPS and 125 MiB/s baseline included (EBS docs) — EBS volumes
- stop dev and test environments nights and weekends — about two thirds or more of their hours (arithmetic on a 168 h week) — non-production
- lifecycle rules for multipart uploads, tiering, old versions — storage line (see Object storage) — buckets
- Naive → standard → state of the art:
- on demand everywhere, no tags · list price, no owner
- tags, budgets, Savings Plan on the floor, Spot for batch · up to 72 % and 90 % off the covered usage
- weekly sort by usage type, unit cost per request or user, architecture fixes for egress and NAT · cost follows usage
Implementation notes
- Tag at creation through Infrastructure as code default tags (team, service, env); untaggable cost goes to a shared bucket reviewed monthly.
- Use the cost and usage report (CUR / BigQuery billing export / Azure cost exports) in SQL; the console hides usage types.
- Unit economics: track cost per request, per user or per GB processed, not just the total.
- Commit in tranches (e.g. quarterly) so a wrong forecast doesn't lock in waste.
- Arm instances: order of 20 % cheaper per vCPU; most Go/Rust/Java/Node services port with a rebuild.
Pitfalls
- Committing to the peak: unused commitment is pure waste and can't be cancelled.
- Free tiers that silently end after 12 months.
- Autoscaling with a minimum that equals the old peak.
- Data transfer between regions for "DR" replicas that nobody tested.
- Optimizing CPU while egress or logging is the real line item; always sort by usage type first.
Implementations
| name | lang | repo | install | verdict | licence | health | as_of | notes |
|---|---|---|---|---|---|---|---|---|
| OpenCost | Go | opencost/opencost | n/a (Kubernetes install via Helm, see docs) | — | — | — | — | cost allocation by namespace, label, workload |
| Infracost | Go | infracost/infracost | brew install infracost | — | — | — | — | price estimate in pull requests |
| Cloud Custodian | Python | cloud-custodian/cloud-custodian | pip install c7n | — | — | — | — | policy rules to stop idle and untagged resources |
| Karpenter | Go | aws/karpenter-provider-aws | n/a (Helm chart, see docs) | — | — | — | — | node provisioning with Spot and consolidation |