Three Layers of AI Infrastructure, Thirty-Three Minutes Apart: Debugging a Multi-Provider AI Orchestration Stack

Before 17:03 on April 25, every part of MockEvalio that called an LLM did it directly — its own HTTP call, its own retry logic, its own assumption about which model to use. By 17:36, thirty-three minutes later, none of that was true anymore. Three commits landed in that window, each one built directly on top of the one before it.

The orchestrator: one entrypoint instead of five

The first commit introduced a single AIOrchestratorService — 479 new lines — sitting between every AI-calling feature and the model itself. It deduplicates identical requests, applies rule-based shortcuts before spending a call on the model at all, routes by how complex the task actually is, enforces per-user limits, and falls back to another path if a provider call fails. Five places that used to call Groq on their own — question generation, evaluation, resume chat, practice plans, job explanations — were rewritten to go through this one entrypoint instead. The same commit adds the first dedicated usage-tracking service in the codebase: a new AIUsageMetricsService, where before there wasn't one.

Prompts stop being strings compiled into the binary

Twenty-one minutes later, the second commit changed what a prompt actually is. Prompt templates moved into database rows — a new PromptTemplate entity, sticky A/B variant assignment, weighted traffic splits, and an admin interface to manage all of it. The orchestrator built minutes earlier was immediately extended to resolve which prompt variant to use and log the outcome. The practical difference: changing what MockEvalio actually says to an LLM stopped requiring a deploy.

Then a cost model, built on usage data that was minutes old

Twelve minutes after that, the third commit added GroqProvider and OpenAIProvider as interchangeable options the orchestrator could route between — the first time OpenAI was a real, first-class inference choice rather than the narrow fallback it had been for embeddings and speech transcription. A new TokenEstimatorService predicts token count and cost before a call runs, with admin-configurable routing weights and telemetry specifically built to compare that prediction against what a call actually cost afterward — a feedback loop meant to make the estimate better over time.

What never got touched again

Neither the router nor the token estimator built in that third commit appears anywhere else in the repository's history. Not a tuning pass, not a bug fix, not a single commit referencing the estimated-versus-actual comparison the telemetry exists to enable. The orchestrator itself does get touched once more, the next day — but only to add tracing spans as part of a broader observability push, not to change how it routes or decides anything.

What I don't know about that half hour

Whether these three layers were one design, executed in order because that's the order they'd already been planned in — or whether finishing the orchestrator made the need for versioned prompts obvious, and finishing that made the need for cost-aware routing obvious in turn — isn't something the commit timestamps can settle. Nothing landed in the three weeks before this either; the gap is real, and I'm not going to guess at what it means. And I don't know whether the estimate-versus-actual telemetry was ever looked at, even once, or whether it's been quietly recording a comparison nobody has read since the day it shipped.

What's certain is the shape: three layers, each one assuming the one below it, built in the time it takes to get through a short meeting, and never revisited since.