When to Build a Single LLM Orchestration Layer

Most codebases that call an LLM provider start the same way: a service that needs a completion calls the provider's SDK directly. It works, and it keeps working, right up until there are five or six call sites doing the same thing slightly differently — different retry logic, no shared way to avoid two identical calls happening seconds apart, no single place to see what a feature is actually costing, and no way to change which model handles a task without a code change and a redeploy.

The signal that it's time

The trigger isn't a fixed number of call sites. It's when you can no longer answer three questions without grepping the codebase: which provider is handling a given task right now, what a given feature costs in aggregate, and whether two call sites are duplicating identical requests. In MockEvalio's case, five call sites — question generation, evaluation, resume chat, practice plans, job explanations — had each grown its own direct integration before this became the actual constraint.

What the layer needs to own

A working version of this needs to own, at minimum: deduplication of identical in-flight requests, rule-based shortcuts that skip the model entirely when a cheaper deterministic answer is available, routing by task complexity, per-user or per-feature usage limits, and provider fallback when a call fails. None of this is provider-specific — it's a control layer that sits in front of whichever provider ends up handling the call.

Prompts as data, not code

A closely related decision: whether prompt text lives as a string compiled into the binary, or as a row a service reads at request time. The practical difference shows up the first time you want to A/B test two phrasings of the same prompt, or fix a prompt that's producing bad output, without shipping a deploy. Versioned prompt entities with weighted traffic splits and usage logging turn prompt iteration into a data change instead of a code change — a meaningfully different operating model once a product has more than a couple of prompts in production.

Making routing decisions cost-aware

The third layer worth building deliberately, not accidentally: a routing decision that accounts for cost before making the call, not after. That means estimating token count and cost pre-call, not just logging it post-call, and treating the estimate itself as something to validate — telemetry that compares estimated cost against actual cost over time is what tells you whether your routing model's assumptions are still true.

The trade-off worth naming honestly

Building this layer is real infrastructure investment before it pays for itself. The failure mode worth worrying about isn't building it too early for a product with two call sites and no cost pressure — it's building the estimation and telemetry half of it and never closing the loop: adding the machinery to compare estimated versus actual cost, then never actually using that data to revisit the routing weights it was built to inform. Instrumentation that nobody looks at is a sunk cost with an ongoing maintenance tax, not a working system. If you build the cost model, the routing weights it informs need an actual owner and an actual review cadence — otherwise skip the estimation layer and keep the simpler version until someone has time to close that loop.

The practical recommendation

Build the orchestration and dedup layer as soon as you have more than two or three call sites doing ad hoc provider calls — that part pays for itself almost immediately in reliability and reduced duplicate spend. Build the versioned-prompt layer once you're iterating on prompt quality often enough that a deploy cycle is the bottleneck. Build the cost-estimation layer last, and only once you're committed to actually reviewing what it tells you — otherwise it's just more code to maintain for a comparison nobody runs.