Taming Alert Noise: One‑Night OpenTelemetry Observability Build

I built full observability into MockEvalio in one night. By the next morning, I was turning parts of it back off.

What got built, in three passes before midnight

The first commit, just before midnight on April 2, wired OpenTelemetry OTLP into both the API and the background worker, pointed at SigNoz, with the deploy pipeline writing the connection details into the server's environment on every deploy. Nine minutes into April 3, a second commit switched logs — not just traces — to export the same way, and added runtime and ASP.NET Core metrics on top. Fourteen minutes after that, a third commit enriched what got sent: instance and environment tags on every signal, exceptions and HTTP status codes recorded directly on spans, and a smoke-test endpoint to confirm the pipeline was actually working end to end.

Three commits, thirty minutes apart from each other, and MockEvalio went from exporting nothing to exporting traces, logs, and metrics from its main process.

The next morning, the workers caught up

There's an eight-and-a-half-hour gap after that third commit — reasonably a night's sleep, though that's an assumption, not something the commits themselves say. When the work picks back up, just before nine in the morning, it extends the same coverage to the three background workers, which until then had no structured logging or tracing of their own. Each one gets real instrumentation — spans around its work, structured logs at the start and end of a cycle, and for the job-matching worker specifically, exceptions recorded directly on the span instead of just written to a log line that might scroll past. Twenty minutes later, another commit bumps the OpenTelemetry packages again and adds a handful of structured fields for Redis and HTTP server activity.

Then the fifth commit that morning turned some of it back down

The last commit of the push, at 9:46, is the one that changes the shape of this story. Entity Framework Core logs every SQL command it issues by default — a routine, well-known amount of noise once you actually turn logging on and watch it. This commit filters that specific log category down to warnings only, and replaces it with a purpose-built interceptor that logs exactly two things: queries that are slow, and queries that fail. Everything else — every ordinary, successful query — stops generating a log line at all. The same commit downgrades what its own message calls "repetitive worker logs" from informational to debug level.

Read plainly, in order: instrument everything, extend it to the workers, and then — within the same morning, the same push — start deciding what wasn't actually worth logging at that volume. I don't have a record of what made that obvious. It could have been watching the actual log stream for the first time and finding it unusable. It could have been a SigNoz ingestion limit or cost figure. It could just as easily have been a five-minute judgment call while reviewing what had shipped a few hours earlier. The commit tells me what changed. It doesn't tell me what I was looking at when I decided it needed to.

Three weeks later, the same theme kept going

The observability work didn't stop there — it picked back up on April 26, twenty-three days later, with a meaningfully different scope. The first push instrumented infrastructure: HTTP requests, worker cycles, SQL queries. This one adds correlation IDs that follow a request across services, and — the real shift — spans and metrics inside the actual product logic: AI orchestration calls, interview sessions, resume pipeline runs, job ingestion. The first phase answered "is the system running." The second one starts answering "what is the system actually doing." Whether that gap was always the plan, or something specific made it worth coming back to, isn't something the commit history tells me either.

What I do know is the order things happened in: build broad coverage first, learn within hours that broad and useful aren't the same thing, and only later go back and instrument what the product itself was actually doing. That sequence feels more honest than "we added observability" as a single, tidy line — and it's the one the commits actually support.