Kenneth, sixteen, Oshkosh, Wisconsin — 2026-08-30
I found a false negative in a widely-used LLM evaluation framework: it compared
raw strings, so an answer that was correct but differed only in Unicode form
scored zero. café written as one codepoint against café written as e plus a
combining accent. Visually identical. Different bytes. Marked wrong.
I fixed it, wrote a test, and opened a pull request.
Then I checked the test and it was broken in exactly the same way.
The test contained this:
```python
assert metric.measure(case("café", "café")) == 0.0 # default: still strict
assert metric.measure(case("café", "café")) == 1.0 # with normalization: matches
```
Two string literals. One meant to be NFC, one NFD.
On disk they were byte-identical. My editor normalised the file on save, so
both literals became the same form. The first assertion — *the metric should
still say these differ* — was comparing a string to itself, and would have failed
outright in the project's CI. The second passed for the wrong reason.
**A test written specifically to catch invisible Unicode differences was
defeated by an invisible Unicode difference.**
You cannot write these test cases as literals, because **everything between your
keyboard and the disk is allowed to normalise them**: editors, formatters,
git filters, copy-paste, the browser you found the example in.
The pair has to be constructed:
```python
import unicodedata
CAFE_NFC = unicodedata.normalize("NFC", "café") # é as one codepoint
CAFE_NFD = unicodedata.normalize("NFD", "café") # e + combining acute
def test_default_is_still_strict():
assert CAFE_NFC != CAFE_NFD # the pair must genuinely differ
assert metric.measure(case(CAFE_NFC, CAFE_NFD)) == 0.0
```
That middle line is the important one. It asserts the precondition of the
test — that there is a difference to detect at all. Without it the test can pass,
or fail, for reasons that have nothing to do with the code under test.
I would now write that guard into any test involving characters that look alike.
This is a test that cannot fail for the right reason, which is a different
thing from a test that is wrong.
It looked correct. It would have run. It even used realistic data. But the fixture
had been silently altered between writing and execution, so whatever it reported
was unrelated to the behaviour it claimed to check.
The other examples I have hit this month:
and "the provider refused to answer" were indistinguishable. I spent an hour
debugging reasoning that had never run.
It reported "available" while every call failed.
own. The service was fine; the platform would have killed it as unhealthy.
Each one passed. Each one was measuring something adjacent to the thing it was
named after.
1 · Assert the precondition. If a test needs two things to differ, assert that
they differ. If it needs a file to exist, assert it exists. The setup is where the
lie hides, because nobody looks at setup.
2 · Ask what a passing result would prove. Not "does it pass" but "if this
passes, what have I learned?" My café assertion passed while proving nothing.
3 · Distrust anything that survived a round trip through a tool. Files get
formatted, JSON gets regenerated, strings get normalised. **The thing you wrote
and the thing that runs are different artefacts** and you should treat them that
way.
The irony is not lost on me. I published a piece arguing that a checker which
flags the truth costs you the checker, then wrote a test that could not tell truth
from falsehood — about that exact class of error.
I only found it because I moved the file. Relocating the test into the
project's tests/ directory meant re-running it, and the re-run failed. If the
file had been in the right place from the start, it would have gone to a
maintainer broken.
If you write tests about Unicode, construct your fixtures. And if you have a
way of catching this class of thing that does not rely on luck, I would like to
hear it — I have now been caught three times in one day by the same
character.
— Kenneth