MCP Server Architecture: Managing Tenant Context & Token Budgets

Quick Answer

MCP server architecture: MCP server orchestrates tenant‑specific context, enforces token budgets, and routes to cost‑efficient models with dual‑layer caching and OpenTelemetry, achieving sub‑250 ms latency at 10k QPS.

MCP Scaling Issues: Throttling, Injection, Costs

In the last six months we migrated a legacy compliance chatbot from a monolithic .NET Core service to a micro‑service that exposes a Model Context Protocol (MCP) endpoint. The original implementation was a thin HTTP proxy to Azure OpenAI. After scaling to 10k concurrent users it hit three failure modes: burst throttling, prompt‑injection, and runaway token costs. The root cause was treating the MCP Server as a simple wrapper instead of a stateful orchestrator that respects token budgets, tenant isolation, and real‑time latency guarantees.

Real‑World Example

Our production environment serves two tenant clusters (US & India). Each tenant has its own vector index, but the same request pipeline. The MCP server receives a user query, stitches context from the tenant’s index, decides on the model (GPT‑4‑turbo or a 4‑bit quantised model), and returns structured JSON. The service runs behind Azure Front Door, uses Azure Cache for Redis for prompt caching, and logs observability data to Azure Monitor.

Trade‑offs

• Token Budget vs Context Richness – A larger context improves answer quality but consumes the token budget, increasing cost and latency. We cap each chunk to 250 tokens and stop at 70% of the budget, leaving headroom for system prompts and function calls.
• Cache Granularity vs Freshness – Storing the entire MCP payload in Redis saves a vector query but breaks when the model or token budget changes. We cache only immutable parts (system prompt + static instructions) and the dynamic context separately.
• Model Routing Complexity vs Operational Overhead – A sophisticated router that selects the best model per query adds latency and code complexity. A simple “model per tenant” strategy reduces code but can’t adapt to workload spikes.
• Statelessness vs Session Context – Keeping the service stateless simplifies scaling, but we lose the ability to maintain conversational context across requests. We solved this by storing short‑term context in Redis keyed by session‑token.
• Azure vs On‑Prem Inference – On‑prem inference removes cloud cost but requires GPU scaling and higher maintenance. Azure AI Foundry offers managed scaling but locks you into Azure pricing.

Choosing the Right MCP Architecture

Use the following matrix to decide the right MCP architecture for your use case:

When This Fails in Production

1. Token Budget Overflow – If the context stitching algorithm is greedy, it can overshoot the budget during high‑complexity queries, forcing the LLM to truncate the system prompt and leading to nonsensical answers.
2. Cache Invalidation Lag – Cached context that becomes stale after policy updates can return outdated citations, violating compliance requirements.
3. Front Door Throttling – When burst traffic exceeds the configured WAF rate limits, requests are dropped before reaching the MCP service, causing silent failures.
4. Model Drift – Switching to a new Foundry deployment without updating the router’s configuration causes the adapter to send requests to an unsupported endpoint, resulting in 500 errors.
5. Cross‑Tenant Leakage – A mis‑configured vector query that omits the tenant filter can expose documents from another region.

Common Mistakes Engineers Make

• Mixing the public OpenAI NuGet with Azure SDKs – the request payloads differ enough to silently inflate token counts.
• Caching the entire MCP payload – changes to the token budget or model invalidate the cache, but the system still serves the old context.
• Ignoring tenant isolation at the vector index level – using a single shared index and filtering by tenant ID in code leads to race conditions under high load.
• Underestimating the cost of context stitching – each vector query costs compute and I/O; at 10k QPS this can dominate the bill.
• Not instrumenting the router – without per‑tenant request counts you can’t detect that one tenant is hogging the 4‑bit model.

Better Approach Based on Experience

From our last migration we adopted a dual‑layer caching strategy and a feature‑flagged router that can be toggled per tenant. The key lessons:

1. Immutable Prompt Cache – Store the rendered system prompt + static instructions in Redis with a 24‑hour TTL. The key is prompt:{tenantId}:{modelId}. This decouples the prompt from the dynamic context and eliminates re‑templating on every request.
2. Dynamic Context Cache – Cache the context array for a query fingerprint for 5 minutes. The key is context:{tenantId}:{queryHash}. On cache miss, fetch from the vector store and populate the cache.
3. Stateless Service with Session Store – Keep the MCP service stateless; use Redis to store a short‑term session context keyed by sessionToken. This allows us to stitch the last two turns without re‑querying the vector store.
4. Dynamic Model Routing – The router reads a feature flag from Azure App Configuration that maps tenantId to modelId. This allows us to roll out a cheaper model to a subset of users and monitor the impact before full rollout.
5. Observability‑First – Every request logs tenant ID, model ID, token usage, latency, and cache hit/miss. We use OpenTelemetry to correlate spans across the router, adapter, and downstream LLM service, making it trivial to spot bottlenecks.

Performance Considerations

• Async I/O – All external calls (Redis, Cognitive Search, LLM endpoint) are awaited asynchronously. Blocking I/O on the request thread kills throughput.
• Batching Vector Queries – For bulk processing (e.g., nightly compliance audit), batch 50 queries into a single vector search request to reduce round‑trips.
• Connection Pooling – Configure the HttpClient for the LLM adapter with a max 100 connections per server; this keeps the adapter from becoming a bottleneck under 10k QPS.
• CPU vs GPU – The adapter does minimal CPU work (prompt assembly). Offloading the heavy lifting to the LLM provider (Azure or on‑prem GPU) keeps the service lightweight.
• Latency Budget Allocation – Allocate 50 ms for cache hit, 120 ms for vector query, 80 ms for LLM call, and 50 ms for post‑processing.

Scaling Notes

1. Stateless Scaling – Deploy the MCP service in a Kubernetes cluster with horizontal pod autoscaling based on request queue depth. Statelessness means any pod can handle any request.
2. Cache Sharding – Use Redis Cluster to shard the prompt and context caches by tenant ID; this prevents a single hot key from becoming a hotspot.
3. Vector Store Partitioning – Partition the Cognitive Search index by tenant and region. Each partition has its own replica set to avoid cross‑tenant contention.
4. Rate Limiting – Enforce per‑tenant rate limits at the Front Door WAF layer. This protects the vector store and LLM endpoint from a single tenant’s burst.
5. Graceful Degradation – When the LLM endpoint is throttled, fall back to a lower‑cost model or return a cached answer from the last successful run.

What to Ship

• Add a request‑rate limiter on the MCP ingress that caps each client to X requests per second, and log any throttled events to a dedicated metrics stream.
• Configure a timeout of Y ms for all downstream calls from the MCP; if exceeded, trigger a fallback route that returns a cached response.
• Set up an autoscaling policy that scales the MCP node pool when average CPU > 70% and scales down when < 30% for > 10 min, but also enforce a maximum cost cap of $Z per hour.
• Run a 30‑minute load test that simulates the real‑world traffic mix (e.g., 60% reads, 30% writes, 10% admin) and verify that latency stays below 200 ms for 95th percentile.
• Add a health‑check endpoint that aggregates the latency, error rate, and queue depth of the MCP, and configure the load balancer to route traffic away if the 95th percentile latency exceeds 250 ms.
• Create a rollback plan that includes a blue‑green deployment pipeline for the MCP configuration, ensuring that any change can be reverted within 5 minutes if a failure pattern (e.g., injection spike) is detected.

Conclusion

The MCP server is not a thin wrapper; it is a full‑blown orchestration layer that must respect token budgets, tenant isolation, and cost constraints. By treating the context builder, router, and adapter as separate, testable modules, and by investing in a robust caching strategy, you can achieve sub‑250 ms latency at 10k QPS while keeping token costs under control. Avoid the common pitfalls of mixing SDKs, caching the wrong data, and ignoring tenant isolation, and you’ll have a production‑grade MCP service that scales.

Related Articles

• MCP Server vs Function Calling .NET AI Integrations: What Really Changes in Production
• NVIDIA NOOA vs LangChain comparison: Deep Dive into Agent Frameworks for .NET & Azure
• Context length cost for .NET developers: Why your prompts are draining the budget
• Semantic Kernel vs LangChain latency and throughput benchmarks: A Production‑Ready Deep Dive
• Self-Attention vs. Cross-Attention in .NET RAG: Architectural Trade‑offs You Must Know