Observability for LLM Apps in ASP.NET Core: Trace First, Metrics

Quick Answer

Observability for LLM Apps in ASP.NET Core: Trace‑first, metric‑second, feedback‑first observability in ASP.NET Core LLM apps balances cost, latency, and model quality.

Missing Observability Contracts Drive Cost Catastrophes

In a world where an LLM call can cost a few dollars per thousand tokens, a single malformed request can turn a profitable feature into a cost‑catastrophe. The symptoms are familiar: a spike in latency, a sudden increase in token usage, and a customer‑reported hallucination that never surfaced in QA. The root cause is not a missing logger; it is a missing observability contract that treats every LLM invocation as a first‑class transaction, with context, metrics, and a feedback loop.

Real‑World Example: A Bilingual RAG Chatbot in a Multi‑Tenant SaaS

We built a chatbot that serves 10,000+ tenants across the US and India. The stack is: API Gateway → ASP.NET Core Chat Service → Vector Store → Semantic Kernel → Azure-openai-service-vs-gpt4-api-for-net-microservices-a-deepdive-for-architects-20260830" class="internal-link">Azure OpenAI. Each tenant has its own data store and quota. In production we saw:

• Average token cost per request: $0.0004, but a single tenant’s mis‑configured prompt pushed the cost to $0.04.
• Latency: 120 ms in dev, 1.5 s in prod during a traffic spike.
• Hallucinations: 3% of responses were flagged by our post‑processing validator, but the root cause was buried in a chain of 7 spans.

Without a trace‑first observability stack, debugging required chasing logs across services, manually correlating IDs, and re‑running the request locally. The time‑to‑resolution was 3–4 hours, unacceptable for a SaaS SLA.

Trade‑offs: Metrics, Tracing, and Feedback Loops vs. Complexity and Cost

• Granular metrics (tokens, prompt length, hallucination flag) give you the data you need to predict cost and detect anomalies, but they add overhead to every request. In high‑throughput environments, even a 1 µs cost per metric can add up.
• Full OpenTelemetry tracing provides end‑to‑end visibility, but requires context propagation, span naming conventions, and a backend that can ingest millions of spans per second. Choosing Application Insights alone limits custom attributes; a self‑hosted Collector + Tempo gives you flexibility at the cost of operational overhead.
• Evaluation loops (heuristics + human review) are essential for agentic AI but introduce latency if you wait for human feedback before a response is sent. The trick is to surface only the most critical failures to reviewers, using automated filters to keep the loop tight.
• Cost vs. observability depth: Each extra span or metric you emit can increase ingestion fees (Azure Monitor, Prometheus, Loki). In a multi‑tenant SaaS, you need to balance the granularity of telemetry against the bill‑to‑customer model.

Scaling Observability Stack for High‑Volume LLM Apps

1. Assess the volume: If you expect >10 M LLM calls per month, a hybrid model (Application Insights for alerts + OpenTelemetry Collector + Tempo for deep traces) scales better than a single vendor.
2. Determine the granularity required: For cost‑control, token‑level metrics are non‑negotiable. For latency debugging, you need sub‑span timing (network vs. inference).
3. Evaluate operational overhead: A self‑hosted Collector + Tempo demands cluster management, backups, and upgrades. If your ops team is small, lean on Azure Monitor and enrich it with custom metrics via the SDK.
4. Plan for multi‑tenant correlation: Always propagate traceparent and add tenantId, conversationId tags. This enables cross‑service analysis without leaking data between tenants.
5. Feedback loop strategy: Start with automated heuristics (JSON schema, regex). Add a lightweight review UI only for high‑impact failures. Automate retraining triggers via Azure Functions or a serverless scheduler.

When This Fails in Production

• Missing context propagation – The trace ends at the HTTP controller, and the LLM span is orphaned. You cannot correlate latency spikes with downstream services.
• Over‑instrumentation in hot paths – Recording a histogram for every token count in a 1 ms request can become a bottleneck. Use sampling or aggregate metrics per tenant.
• Inadequate sampling policies – If you sample too aggressively, you lose visibility into rare but critical failures; if you sample too conservatively, you drown in noise and hit ingestion limits.
• Unencrypted telemetry in transit – In a multi‑tenant environment, you must enforce TLS for all telemetry endpoints to avoid data leaks.

Common Mistakes Engineers Make

• Relying solely on application logs – Structured logs are great, but they lack the causal chain that tracing provides.
• Ignoring the cost of metrics – A naive implementation that records every token count as a separate histogram can exceed Azure Monitor’s free tier quickly.
• Failing to instrument third‑party SDKs – The Azure OpenAI SDK does not emit traces by default. You must wrap or decorate the client.
• Not correlating business identifiers – Without tenantId or conversationId tags, you cannot slice telemetry by tenant or user, leading to blind spots in multi‑tenant analytics.
• Treating evaluation as a one‑off – Without an automated retraining pipeline, you’ll accumulate stale data and never improve model quality.

Better Approach Based on Experience

In production, I adopt a trace‑first, metric‑second, feedback‑first strategy:

• Instrumentation: Wrap every Semantic Kernel call in an Activity. Use ActivitySource with a consistent namespace (e.g., MyCompany.ChatService). Inject traceparent into the Azure OpenAI HTTP client via a delegating handler.
• Metrics: Emit a Histogram for llm.tokens and llm.latency_ms per tenant, but sample 1% of requests and aggregate the rest. Use MeterProvider to push to Azure Monitor or Tempo.
• Evaluation Loop: Run a nightly batch job that scans the audit store for failed evals. If json_valid < 95% over the last 1,000 requests, trigger an Azure Function that packages the failing prompts and sends them to a fine‑tune job. Keep the function idempotent and retry‑safe.
• Observability Backend: Deploy an OpenTelemetry Collector in a stateful set, backed by Tempo (for traces) and Loki (for logs). Use k8s‑prometheus‑operator to expose metrics to Grafana. For cost control, set retention to 14 days for traces and 7 days for logs.
• Security & Compliance: Mask sensitive prompt content before logging. Store raw prompts only in a secure, encrypted audit table with row‑level access controls.
• Scaling Notes: Each LLM call adds a span. In a 10 k TPS environment, you need a Collector that can ingest 1 M spans per second. Scale horizontally, use autoscaling, and backpressure the incoming request queue if the Collector is saturated.

Performance Considerations

• Span creation overhead – Activity creation is cheap (<1 µs) but recording many attributes can be expensive. Keep tags minimal; use SetTag only for high‑value data.
• Metric aggregation – Use Histogram buckets that align with your SLA thresholds (e.g., 100 ms, 500 ms, 1 s). Avoid per‑token metrics; aggregate per request.
• Backpressure – If the Collector is overwhelmed, drop low‑priority spans using sampling. Do not let telemetry block the main request path.

Scaling Notes

• Collector scaling – Run the Collector as a StatefulSet with 3 replicas; use a load balancer to distribute spans. Monitor QueueLength metrics to trigger scaling.
• Log ingestion – Loki can store 1 TB of logs for 7 days at $0.20/GB/month in a managed cluster. For high‑volume services, partition logs by tenant to avoid hot keys.
• Tracing backend – Tempo can ingest 10 M spans per second per node. In a 5‑node cluster, you get 50 M spans/s. Adjust retention to keep only the last 7 days for cost control.
• Cost monitoring – Set up alerts on Azure Monitor for DataIngested and DataStored. Use tags to attribute cost per tenant and adjust quotas accordingly.

What to Ship

• Define an explicit observability contract for every LLM endpoint: list required trace attributes (e.g., tenantId, requestId, modelVersion), evaluation metrics (e.g., token‑level latency, accuracy scores), and feedback payloads, then add a compile‑time check that the contract is implemented in ASP.NET Core middleware.
• Instrument the bilingual RAG chatbot with OpenTelemetry in ASP.NET Core, adding a custom activity that records the vector‑store lookup time, the selected source documents, and the tenant ID; enable propagation of the activity ID across all gRPC / HTTP calls.
• Set up an automated evaluation pipeline that runs every 10 minutes, pulls the last 100 requests per tenant, calculates BLEU/ROUGE scores against a gold‑standard, and writes the results to a tenant‑scoped Prometheus metric; alert if the average score falls below the SLA threshold.
• Configure adaptive sampling in Jaeger/Tempo: sample 1 % of all requests by default, but increase the sample rate to 20 % for any request whose latency exceeds 2 s, and drop traces that exceed the configured cost‑budget per tenant.
• Deploy a tenant‑isolated observability stack (Prometheus, Grafana, and Tempo) with a separate namespace per tenant; set retention to 30 days for metrics and 7 days for traces, and schedule nightly compaction jobs to purge data that has exceeded its TTL.
• Add a health‑check endpoint (/health/observability) that verifies connectivity to the tracing collector, the metrics endpoint, and the evaluation database; fail the deployment if any of these services return a non‑200 status.

Conclusion: Observability is Not an Add‑On, It’s the Backbone of a Robust LLM Service

For a senior architect, the choice of observability stack is a strategic decision. It determines how quickly you can detect anomalies, how you manage cost, and how you evolve your models. By treating each LLM call as a first‑class transaction, instrumenting it with context, and feeding back the results into a retraining loop, you build a system that not only scales but also learns from itself. The trade‑offs are clear: more telemetry means more operational overhead and cost, but the upside is a predictable, auditable, and continuously improving AI service.

Related Articles

• Designing a Multi-Tenant KV-Cache Layer in ASP.NET Core for Scalable Inference Serving
• LLM Cost Control in .NET: Debugging Billing Surprises in Production
• Azure OpenAI Service vs GPT‑4 API for .NET Microservices: A Deep‑Dive for Architects
• Building a Production Agent Harness in ASP.NET Core: The Five‑Layer Blueprint
• Using evals as release gates for LLM changes in .NET CI/CD pipelines