The Follow-Up Question That Always Started With "What Would You Sacrifice"

The Follow-Up Question That Always Started With "What Would You Sacrifice"

August 17, 2026 4 min read
Build in Public MockEvalio AI Quality / LLM Output Diversity

Somewhere in MockEvalio's interview flow, the AI kept reaching for some version of "what would you sacrifice to ship this faster?" I don't have a transcript proving that — no log of every follow-up question the model ever generated. What I have is what I did about it: on March 27, I wrote code that explicitly tells the model not to open a follow-up with that phrase, or with its FAANG-persona cousin, "how would you handle the trade-off." Naming a phrase specifically enough to ban it twice, in two separate prompts, in the same commit, is a strong sign it was the actual problem. It isn't a recording of the problem. I want to be honest about that distinction rather than claim more than the evidence gives me.

What the old logic actually did

Before this commit, the rule for whether to ask a follow-up question was simple to the point of being blunt: if the candidate's answer scored 70 or above, ask a follow-up. Every time. Weak answers got a follow-up too, for a different reason — to guide the candidate — but strong answers had no other path. Answer well, and the next thing you saw was always another question probing the same answer, generated by the same prompt, shaped by the same instructions. Repetition wasn't a side effect here. It was what the code was designed to do.

Three changes, one commit

The fix wasn't a single change. It was three, aimed at three different layers of the same problem.

First, a probability gate. A new setting — Interview:StrongAnswerFollowUpProbability, defaulting to 0.35 — means a strong answer now gets a follow-up roughly a third of the time. The rest of the time, the system pulls a genuinely new question from the bank, from RAG, or fresh from Groq instead. Weak answers are untouched; they still always get a guiding follow-up. The fix targeted the specific case that was repeating.

Second, the prompts themselves got a rule added, word for word: "Do not default to 'What would you sacrifice...' or 'How would you handle the trade-off...'. One sentence. No preamble." That instruction went into every follow-up prompt, and the two interviewer personas most likely to lean on those exact phrases — FAANG and Startup CTO — got their instructions rewritten separately, with the same two phrases named again and a list of alternative angles to rotate through instead: invariants, failure modes, load testing, incident response, API contracts, cost estimates.

Third, the sampling temperature for follow-up generation went from 0.4 to 0.72 — nearly double. Lower temperature makes a language model more likely to reach for its most probable continuation every time, which is exactly how you get the same phrase back repeatedly. Raising it doesn't guarantee variety, but it removes some of the pressure toward the single most likely answer.

None of these three changes would have fully solved the problem alone. A probability gate reduces how often a follow-up fires, but says nothing about what that follow-up says when it does. A phrase ban stops two known phrasings, but not a hundred others like them. A temperature increase adds variety, but not control over what that variety actually looks like. Together, they cover more of the problem than any one of them does by itself.

What I don't actually know

I don't know what specifically made me notice this — whether it was testing the product myself across several sessions, something a tester said, or just running it once and hearing the same phrase come back. There's no note anywhere describing that moment. I also don't know whether 0.35 and 0.72 were chosen after any real testing or were reasonable numbers to start with. And I've never gone back to check whether this actually reduced repetition in practice — nothing in the repository touches this code again after this commit.

I know exactly what the fix changed. Whether it actually fixed the thing I built it to fix isn't something I can answer from the commit history alone.