Capacity Estimation in System Design: A Kubernetes Autoscaling Case

Capacity Estimation in System Design: A Practical Walkthrough for Scalable Architecture

Quick Answer

Learn why capacity estimation in system design is critical, how to model workloads, choose scaling patterns, and avoid costly pitfalls with real‑world case studies and actionable tools.

CPU Misestimation Drives Cost Surges

In a production system that’s already hit the 10‑million‑request‑per‑second mark, a single off‑by‑one in the CPU‑per‑request figure can turn a $10k/hour bill into a $200k bill. The root cause isn’t a lack of data; it’s the absence of a disciplined, end‑to‑end mental model that ties business load to concrete resource consumption. Below is a hardened approach that I’ve used in a multi‑region SaaS platform, with a focus on what actually breaks in production, the most common missteps, and a pragmatic path forward.

Opinion: I’ve seen teams spend months on “capacity planning” spreadsheets that never get validated. The real trade‑off is between the upfront cost of building a telemetry pipeline and the downstream cost of an outage. In most cases, a modest investment in observability pays for itself in avoided downtime and a clearer budgeting cadence.

Real‑World Example: The Flash‑Sale Crash

Last quarter a flagship e‑commerce client launched a flash‑sale that pushed traffic from 6k QPS to 18k QPS in under a minute. The initial capacity plan had provisioned 8 vCPUs per pod, assuming 12ms CPU per request. The 99.9th‑percentile latency SLA was 150ms.

At 18k QPS, each pod hit 95% CPU, but the autoscaler only reacted after 5 minutes because it was tuned to a 30‑second average.
The burst caused a cache‑miss storm; Redis cluster saturated, spilling to disk I/O, and the latency ballooned to 350ms.
Result: 1.2× SLA violation, a $15k spike in spend, and a 12‑hour outage to rebuild the cache.

What I would have done differently: pre‑warm the cache during the first 5 minutes of a known flash‑sale window and use a burst‑aware autoscaler that reacts to a 1‑minute rolling average. Also, decouple the cache layer from the compute pool to avoid shared IOPS contention.

Trade‑offs in Capacity Planning

Static vs. Dynamic Models
- Static: quick, but ignores burstiness and background jobs.
- Dynamic: more accurate, but requires continuous data pipelines and simulation.
Vertical vs. Horizontal Scaling
- Vertical: lower network hops, simpler but hits cloud limits quickly.
- Horizontal: linear cost, better fault isolation, but adds inter‑node latency and sharding complexity.
Headroom vs. Cost
- 30% buffer protects against unseen spikes but can double cost if left idle.
- Dynamic autoscaling with CPU+memory triggers reduces waste but may miss sudden network bottlenecks.
Granularity of Metrics
- Per‑request CPU and DB latency give the best fidelity.
- Aggregated metrics hide per‑path variance; they’re insufficient for fine‑tuned autoscaling.

Trade‑off highlight: When you need to support a 99.99th percentile SLA, the cost of per‑request telemetry is justified; if your SLA is 95th percentile, aggregated metrics may suffice and save on storage.

When This Fails in Production

Unmodeled Cache Misses – A 5% drop in hit ratio can double DB load during a surge.
Autoscaler Lag – If the rule only looks at average CPU over 5 minutes, a 1‑minute spike can push latency past SLA before new pods spin up.
Background Jobs Sharing Pools – Nightly ETL processes that run on the same node pool can starve web traffic during peak hours.
Resource Contention Across Services – A microservice that logs to a shared file system can saturate IOPS, pulling down unrelated services.

What to avoid: Treating a single metric as the sole scaling trigger. In practice, the most common failure is ignoring the coupling between DB latency and cache behavior, which can create a feedback loop that the autoscaler never sees.

Common Mistakes Engineers Make

Assuming a fixed cache hit ratio and never validating it against production traffic.
Hard‑coding thread pool sizes in .NET without accounting for async I/O spikes.
Using a single autoscale rule that only monitors CPU, ignoring memory fragmentation and GC pauses.
Treating “30% headroom” as a magic number without tying it to observed variance in the 99.9th percentile.
Ignoring the cost of network egress when scaling globally; a 10% increase in cross‑region traffic can double egress fees.

Additional pitfall: Over‑optimizing for the “average” request path and under‑investing in the edge cases that actually drive the SLA. The real cost is often in those edge cases.

Better Approach Based on Experience

Adopt a simulation‑driven, data‑centric workflow that iterates weekly:

Collect granular telemetry – per‑request CPU, DB latency, cache hit/miss, GC pause, thread pool depth, and network I/O. Use dotnet-counters, PerfView, and Azure Monitor.
Model traffic spikes – Fit a log‑normal distribution to QPS over the last 30 days, then run a Monte Carlo simulation to derive the 99.9th percentile CPU requirement.
Translate to infrastructure – Convert CPU cores to VM sizes per region, factoring in the VM’s CPU pinning and memory overhead. Use Azure’s VM Size Recommendations API to validate.
Configure multi‑metric autoscaling – Set CPU > 70% *and* memory < 500MB average over 3 minutes, with a 5‑minute cooldown. Include a “spike” rule that reacts to sudden increases in request count.
Validate with staged load tests – Run a k6 script that ramps from 10k to 20k QPS, monitoring real‑time metrics. If latency breaches the SLA before autoscale kicks in, tighten the rule or add more headroom.
Automate drift detection – When observed CPU per request deviates >10% from the model for 3 consecutive intervals, trigger a pipeline that re‑runs the simulation and updates the autoscale config.

Beware of the “simulation‑bias” trap: if your telemetry is stale or your model ignores a new microservice, the simulation will under‑estimate. Keep the telemetry pipeline lightweight but real‑time.

Performance Considerations & Scaling Notes

CPU per request in a .NET API is highly dependent on GC pressure; model GC pause as a separate variable and include a 20% overhead in the simulation.
Network egress cost can eclipse compute cost at scale; keep a cache layer in the same region to reduce cross‑region traffic.
When sharding a relational database, remember that each shard’s CPU can become the bottleneck; monitor per‑shard latency and re‑balance if necessary.
For global traffic, use Azure Front Door’s latency‑based routing but keep a single cache cluster per region to avoid cache coherence traffic.
In Kubernetes, use horizontalpodautoscaler with resource: cpu and memory metrics, but also expose a custom metric (e.g., request_latency_ms) to trigger scaling on latency spikes.

Key decision: If your latency SLA is tighter than 150ms, add a custom metric to the HPA; otherwise CPU/memory alone may be sufficient.

Decision Guide: When to Choose Which Strategy

Scenario	Recommended Strategy	Key Decision Criteria
Short‑term spike (e.g., flash sale)	Horizontal scaling with a burst‑aware autoscaler + cache pre‑warming	Peak QPS > 2× average; cache hit ratio < 90%
Long‑term growth (steady 10% month‑over‑month)	Vertical scaling to higher‑core VMs + right‑sizing reviews	CPU utilization < 60% for >90% of time; memory < 70%
Microservices with high RPC latency	Introduce a second cache layer + request batching	Per‑hop latency > 10ms; end‑to‑end SLA < 250ms
Multi‑region compliance requirement	Deploy read replicas per region + global traffic manager	Latency SLA < 100ms; data residency rules
Cost‑sensitive environment	Use spot instances + scheduled batch jobs off‑peak	Workload can tolerate 5‑minute downtime; budget < $5k/month

When you’re in a regulated industry, the “multi‑region compliance” row is a hard rule; you cannot trade latency for compliance. In contrast, in a consumer app where latency is less critical, you can push more into a shared pool to shave costs.

What to Ship

Validate CPU capacity with a synthetic workload that mirrors production request mix and run it at 2× expected peak traffic; record CPU usage, memory, I/O and compare against budgeted resources.
Configure autoscaling policies that trigger at 70 % CPU utilisation and test the scaling loop with a simulated traffic surge to confirm instances spin up within 30 s and service latency stays below SLA.
Add a hard cap of X concurrent requests per instance in the load balancer and enforce it with a rate‑limiter; verify that the cap prevents CPU oversubscription during flash‑sale style spikes.
Store the capacity estimate, assumptions, and the validation results in a shared design document; link it to the deployment pipeline so that any change to traffic assumptions requires a formal review.
Create a “capacity review” step in the CI/CD pipeline that automatically re‑runs the CPU test against updated code and fails the build if utilisation exceeds the 80 % safety margin.
Monitor the actual CPU utilisation of production instances against the estimated peak and generate a monthly report; if the average stays below 60 % for 3 consecutive months, consider right‑shifting resources to reduce cost.

Conclusion

Capacity estimation isn’t a one‑off calculation; it’s an iterative, data‑driven discipline. By treating every assumption as a testable hypothesis, validating with real telemetry, and automating drift detection, you can avoid the most common production failures and keep your budget under control. Remember: the real cost of a mis‑estimated capacity isn’t just the extra bill – it’s the lost uptime and degraded user experience.

Validate every new service against the simulation pipeline.
Keep autoscale rules lean – avoid over‑engineering with too many metrics.
Monitor cache hit ratios as a first‑level SLA guard.
Automate drift alerts so you never ignore a 10% shift in CPU per request.
Review cost vs. headroom quarterly; the 30% rule is a starting point, not a target.