The Logs Were Missing. So Was the Deploy Pipeline's Connection to the Server — and Nothing Said So.

MockEvalio's backend already exported logs, traces, and metrics to SigNoz over OTLP — configured months earlier, secrets already in place. The report was simple: no logs showing up. The actual cause turned out to be three separate, unrelated things, and the first mistake was assuming there'd be one.

Problem one: the pipeline could fail silently

Nothing in the codebase surfaced an OTLP export failure anywhere — not in application logs, not in container output, nowhere. A wrong endpoint, a bad protocol, a rejected header: any of it would fail with zero visible trace. Fixed by forwarding the OpenTelemetry .NET SDK's own internal Warning-and-above diagnostics to stderr, and by wiring up a startup smoke-test log — three tagged sample entries emitted on boot — that had existed in the code for months but had no way to be switched on in production without hand-editing the server's .env over SSH.

Problem two: the deploy pipeline itself, mid-fix

Wiring that toggle into CI (a GitHub Actions repo variable, a workflow_dispatch trigger, so it could be flipped and redeployed without touching the server directly) meant a routine merge to main. That merge's own auto-triggered deploy failed — not on anything related to the fix, but on dial tcp ***:22: i/o timeout: GitHub Actions couldn't open an SSH connection to the VPS at all. Every deploy earlier that same day had succeeded. The live site itself was still up and responding in about a tenth of a second when checked directly. A retry succeeded outright — confirming it was a transient connectivity blip, not a server outage, but one that had gone completely unnoticed because nothing had been watching deploy outcomes either.

Problem three: the toggle still hadn't gone out

A second deploy, meant to carry the sample-log toggle live, failed differently — GitHub refused to even start the job: "recent account payments have failed or your spending limit needs to be increased." Unrelated to the code, unrelated to the server, purely a GitHub Actions billing gate on the account running the pipeline. Fixed outside the codebase entirely, then the same deploy re-run succeeded clean.

What actually confirmed the fix

Along the way, the SigNoz OTLP endpoint and ingestion key turned out to need correcting too — region-specific values from SigNoz's own onboarding flow, not the placeholder-shaped values that had been in place. Once endpoint, key, and the deploy pipeline were all actually correct, the sample-log smoke test became the real verification step: three tagged log lines, on demand, independent of whether real traffic was generating anything to look at.

What I still don't know

Whether the SSH timeout was a one-off network blip or the first sign of something that will recur is something only more deploys over time will show — one successful retry doesn't rule out a pattern, it just fails to prove one yet. What this event does show clearly: "the logs are missing" can mean the export pipeline is broken, the deploy pipeline never reached the server, or the CI account itself is blocked — three failure domains that look identical from the outside and require completely different fixes, and conflating them is how a debugging session goes long for no reason.