Mykyta Bulatnikov
← All writing

A check that always passes scores 100%

I built a guard that verifies every number an LLM states against the query it came from. Its score was perfect, and it was broken. How a deliberately lying stub found the bug.

I built a guard for an LLM that answers questions over a metrics warehouse. Its job is simple: every number the model puts in a sentence has to appear in the query result the sentence came from. If a number is not there, the sentence is not allowed out.

On the first eval run, grounding came back at 100%. Every stated number verified. I nearly moved on.

A check that always passes also scores 100%. The score could not tell a guard that works from one that does nothing, so I had to find out which one I had.

Make something lie to it

The fix for an unfalsifiable score is to feed the guard answers that are wrong on purpose and measure how many it catches. I wrote two stubs standing in for the model, each lying in a different way:

  • The hallucinator states a number that is nowhere in the data.
  • The misattributor states a number that is in the data, but from the wrong month.

These are the two failures that matter in practice, and they do not look alike.

how the stub liesmembership checkscoped to the named month
a number that is nowhere in the data100% caught100% caught
a real number, from the wrong month0% caught90% caught

The first row is the easy one: an invented number is absent from the result, so any check finds it. The second row is the finding. A real number attached to the wrong month passes a “does this number appear in the result?” check every single time, and it does not look invented to a human reader either. Only a check scoped to the month the sentence names catches it. The remaining 10% are months that happen to share a value, which no method can tell apart; that is a property of the data, not a bug with a fix.

The bug the stub found

Before the table looked like that, the hallucinator was getting through. Not sometimes: always.

The cause was one character in the number-matching pattern. To avoid matching inside a longer number, the pattern ended with a negative lookahead that excluded a following dot:

# before: refuses a number followed by a full stop
r"-?\d+(?:\.\d+)?%?(?![\d.])"

# after: refuses only a number that continues
r"-?\d+(?:\.\d+)?%?(?!\d)"

That also meant it never matched a number followed by a full stop, which is the last number in almost every sentence anyone writes. The checker looked at “Active accounts were 168 in June.” and reported no numbers found. With nothing to disprove, it passed the sentence.

My unit tests were green, because I had written them with numbers in the middle of sentences. The happy path never touched the failing case. The only thing that exposed it was something actively trying to get a wrong number past the guard.

What I took from it

Test a guard with something that is trying to beat it. Tests written by the person who wrote the check share that person’s blind spots. A stub that lies is an adversary with no such blind spots, and it is cheap to build: two small functions.

A perfect score is a reason to look harder. A check that finds nothing to check has not passed. It has not run. Treat “no numbers found” as a state worth noticing, not a clean result.

Gate the adversarial numbers in CI, not the headline. In the repo, both catch rates are asserted: if an invented number ever survives, or misattribution catching drops below 90%, the eval exits non-zero. The 100% grounding figure on its own would have stayed green through the whole bug.

Report the spread. The model’s routing score across four runs ranged from 60% to 80%. A single eval run is not a measurement, and the guard’s numbers deserve the same suspicion as the model’s.

The full project, with the registry, the MCP server and the eval, is on GitHub, and there is a short case study of how it fits together.