I Built a System to Grade My AI Grader. I Never Gave It Anything to Grade Against: The Missing Benchmark for an AI Interview Evaluator

I built a system to check whether MockEvalio's AI evaluator scores answers accurately. Comparison logic, deviation logging, a threshold-based warning, a periodic calibration worker, an admin dashboard. Every piece of it works. As far as I can find, it has never once had anything to actually check. It shipped at 01:08 on April 26, and it's been sitting complete and unused since.

How it's supposed to work

The mechanism itself is simple. A table of ground-truth entries holds a question paired with an expected score. When a live interview question comes in, the system normalizes its text and looks for a matching row. If it finds one, it compares the AI's actual score to the expected one, logs the deviation, and logs a warning if that deviation crosses a threshold — 1.5 points by default. A background worker runs this comparison periodically across up to 200 ground-truth entries. An admin endpoint summarizes average deviation, accuracy percentage, and the worst-performing prompts.

If none of that setup is in place, though, the code does exactly what it's written to do: it returns immediately, and logs nothing.

What I checked, not assumed

There's no seed data for the ground-truth table anywhere in the commit that introduces it. There's no migration that populates it. There's no endpoint — not in this commit, not anywhere later in the repository's history — that lets anyone add a ground-truth entry. The admin controller only reads; it has no way to write. I checked forward through every commit after this one, specifically for this, rather than assuming the absence meant anything. The table exists. The comparison logic exists. The warning threshold exists. The one thing that was never built is a way to put anything into the table the whole system depends on.

Twelve minutes later, a second commit changed how evaluations get explained — structured strengths, weaknesses, missing concepts, and improvement steps instead of a bare score, validated and persisted so the interview UI could show a real breakdown. That part works independent of the ground-truth gap; it doesn't need comparison data to run.

The rest of that morning

After a gap of around nine and a half hours — reasonably overnight, not something the commits confirm — three more systems landed within an hour of each other. A skill graph that propagates proficiency updates through related skills after an interview, so improving at one thing can partially credit something related. A difficulty service that tracks a rolling average of a session's last three scores and adjusts question difficulty and follow-ups to behave, in the commit's own words, "more like real interviewers." And the capstone: re-interviews get linked back to the session that prompted them, an improvement service tracks score progression and which weak topics persist across sessions, and a new endpoint recommends what to do next — including a specific option to start a re-interview seeded from exactly what someone got wrong last time. A progress-timeline view followed fifteen minutes after that, charting interview, skill, and readiness scores together with trend detection.

That capstone is the actual mechanism behind what I described on the first day of this log — an evaluation someone could actually use to improve, not just a score and nothing after it. This is where that stopped being a sentence and became an API: a specific session, a specific weak topic, a specific button that starts a new interview built around exactly that.

What I don't know

Whether leaving the ground-truth table empty was a conscious sequencing choice — build the measurement system first, populate it once there's time — or something that simply never made it back onto the list once the rest of that morning's work took priority, isn't something the commits settle. I don't know what happened in the overnight gap either. And I don't know whether any of the specific numbers in this day's work — the skill-propagation weights, the three-answer rolling window, the deviation threshold — were chosen from anything more than a reasonable first guess.

What I can say plainly: the part of MockEvalio most directly meant to answer "is the evaluation actually good" was built complete, and has spent every day since with nothing to compare against.