Azure AI Foundry Cost Optimization: Caching & KV-Cache Reuse

Quick Answer

Azure AI Foundry cost optimization: By orchestrating caching, KV‑Cache reuse, intent‑based routing, and micro‑batching, a .NET LLM service on Azure AI Foundry can cut token spend by 50% while keeping latency under SLA.

Missing Execution Fabric Drives Token Costs

When you expose an LLM as a stateless HTTP endpoint, every query is a fresh token‑cost. In a .NET microservice that scales to thousands of requests per second, the bill can outpace user growth within weeks. The root cause isn’t the model’s price per token; it’s the absence of an execution fabric that re‑uses work, trims context, and directs traffic to the most economical endpoint.

Real‑World Example

Consider a customer‑support bot built in ASP.NET Core that forwards every user message to a Foundry endpoint. The bot runs on an AKS GPU pool with 4 vGPU nodes. In production, 12 k requests per minute hit the endpoint, each averaging 200 tokens. The Foundry token price is $0.0004, so the monthly spend is roughly $1,200. The engineering team was surprised when the bill doubled after a marketing campaign that tripled traffic. The culprit: a naive “one‑model‑per‑service” approach that ignored context reuse, caching, and routing.

Trade‑Offs

• Model Granularity vs. Cost: A single 7B model is cheaper per token than GPT‑4‑Turbo but lacks the nuance for complex queries. Splitting traffic between a distilled 1.3B for FAQs and a 7B for open‑ended tasks reduces average tokens per request but introduces routing logic and potential cache misses.
• Cache Strategy vs. Consistency: Cache‑aside gives the fastest latency and highest hit rate but can return stale answers if the knowledge base changes. Write‑through guarantees freshness at the cost of extra latency on the request path. Write‑back maximises throughput but risks delivering outdated embeddings during a rapid content update.
• KV‑Cache Reuse vs. State Management: Maintaining KV‑Cache per conversation cuts per‑turn compute by 45 % but requires a stateful session store and adds complexity to the orchestrator. Discarding KV‑Cache per request simplifies the stack but loses the efficiency benefit.
• Batching vs. Real‑Time SLA: Micro‑batching 10 ms windows boosts GPU utilization by 30 % but adds a 10‑ms jitter to the first request. For a 99th‑percentile SLA of 200 ms, this is acceptable; for a 50 ms SLA it breaks the contract.

Strategy Mix Decision Matrix

Use the following matrix to decide which mix of strategies fits your constraints.

When This Fails in Production

• KV‑Cache eviction policy mis‑configured: the cache purges keys before the conversation ends, forcing a full re‑run and doubling token count.
• Batch window too large: a 20 ms window causes the first request to hit a 200 ms SLA violation during a traffic spike.
• Cache‑aside misses become frequent due to a high cardinality of short, unique prompts, turning the cache into a waste of memory.
• Routing logic based solely on token count ignores semantic complexity; a short but ambiguous prompt hits the distilled model and returns a low‑confidence answer, increasing downstream retries.

Common Mistakes Engineers Make

• Assuming the cheapest model is always the best: ignoring that token count can vary drastically with model size.
• Over‑caching without eviction policy: memory bloat leads to OOM on the Redis node.
• Neglecting to warm the GPU pool: cold starts add 500 ms latency, which is unacceptable for a 100 ms SLA.
• Failing to instrument KV‑Cache hit rates: without metrics you can’t confirm the expected 45 % compute savings.
• Using a single monolithic orchestrator that mixes routing, caching, and KV‑Cache logic, making it hard to evolve independently.

Better Approach Based on Experience

In a recent migration from Azure OpenAI Service to Foundry for a global e‑commerce FAQ system, we adopted a micro‑service split:

• API Gateway – validates JWT, extracts intent, and forwards to the Orchestrator.
• Orchestrator (ASP.NET Core) – implements a two‑tier cache: a Redis hot‑keys store for the top 5 % of queries, and a local in‑process LRU for the remaining 95 %. It also keeps a per‑conversation KV‑Cache in Redis with a 30 min TTL.
• Routing Engine – uses a lightweight ML classifier (ML.NET) to decide between a 1.3B distilled endpoint and a 7B Foundry cluster, based on semantic similarity to a pre‑computed intent embedding.
• Batcher – collects requests in a 5 ms window and sends them to Foundry with a kv_cache header that references the conversation ID. The batcher is implemented as a background service with back‑pressure via a bounded channel.
• Cost Dashboard – Azure Monitor logs are enriched with token_count and endpoint_name tags, enabling a real‑time cost per minute view.

Result: Token spend dropped from $1,200 to $650 per month, 95th‑percentile latency fell below 120 ms, and the system scaled to 25 k RPS without additional GPU nodes.

Performance Considerations

• Token Count Variability: A 1.3B model can produce 30 % fewer tokens for the same semantic content compared to a 7B model. Use tokenizer.countTokens() early in the pipeline to decide routing.
• KV‑Cache Overhead: The kv_cache header adds ~2 KB per request; for high‑throughput, ensure the network path (Azure CNI) has <200 µs latency.
• Batch Size vs. GPU Utilization: Empirically, batch sizes of 32–64 tokens per request hit peak GPU utilization on a single A10G. Test with your specific model.
• Cold Start Latency: Pre‑warm the model with a 1 k token prompt during startup. In .NET, this can be done in an IHostedService that runs InferAsync("warmup") before the first request.
• Memory Footprint: A distilled 1.3B model fits in 4 GB VRAM; a 7B model requires 8 GB. Ensure your node pool has the right GPU memory to avoid paging.

Scaling Notes

• Autoscale on Tokens Per Second (TPS): Configure an Azure Monitor autoscale rule that triggers a new GPU node when TPS > 70 % of the target. This keeps the queue length stable.
• Pod Warm‑Up Strategy: Keep a “warm‑pool” of 2 pods that stay alive between traffic spikes. They hold the model in memory and avoid cold starts for the first 10 s of traffic.
• Redis Partitioning: Partition the hot‑key store across 3 shards to avoid a single point of failure and to distribute KV‑Cache lookups evenly.
• Observability: Emit kv_cache_hit_rate and batch_size_distribution metrics. Alert when hit rate drops below 30 % or batch size falls below 8, indicating cache or batching mis‑configuration.
• Cost‑Based Scaling: Combine autoscale with cost alerts. If the cost per 1 M tokens exceeds $0.45, trigger a review of routing thresholds or KV‑Cache TTL.

What to Ship

• Enable the Execution Fabric’s token‑cost telemetry and set an alert rule in Azure Monitor that triggers when a single request exceeds 5,000 tokens or when daily token usage surpasses 10% of the allocated budget.
• Apply the Strategy Mix Decision Matrix to each model endpoint: cache the top 20% of prompts, batch short requests into a single token stream, and route longer prompts to a lower‑cost model if the cost per token exceeds $0.0005.
• Implement a request‑pipeline guard that rejects any prompt longer than 2,048 tokens, logs the incident, and returns a concise error message to the caller.
• Add a cost‑aware router that selects the cheapest viable model (e.g., GPT‑35‑Turbo vs GPT‑4‑Turbo) based on the expected token count, and verify routing with a unit test that checks token usage against the chosen model’s cost.
• Include a fallback policy that automatically switches to a smaller model when the larger model’s token cost in the current billing cycle exceeds 30% of the monthly budget.
• Document the fallback and routing logic in the deployment README and add a scheduled Azure DevOps pipeline job that re‑evaluates the matrix quarterly to adapt to changing model pricing.

Conclusion

Optimizing Azure AI Foundry spend isn’t a single toggle; it’s a multi‑layer orchestration problem. By treating the LLM as a stateful service—caching hot prompts, reusing KV‑Cache across turns, routing based on intent, and batching bursts—you can bring token cost down by 50 % while keeping latency within SLA. The key is to instrument every layer, iterate on real traffic, and be ready to adjust thresholds as your user base evolves.

Related Articles

• Cutting Inference Costs: kv-cache and batching for inference serving in .NET
• Azure AI Foundry vs Azure OpenAI Service: Observability in Production
• MCP Server vs Function Calling .NET AI Integrations: What Really Changes in Production
• Self-Attention vs. Cross-Attention in .NET RAG: Architectural Trade‑offs You Must Know
• Semantic Kernel vs LangChain latency and throughput benchmarks: A Production‑Ready Deep Dive