Six of Nine Bugs Did Not Exist

We ran a planned revision of our name database. The plan had nine work items, each one a defect to fix. Six of them described defects that were not there, or work that had already been done.

Not one of the six came from a bad measurement. Every single one came from a paragraph in our own documentation.

That is the finding, and it is more uncomfortable than the usual "we found bugs in our data" confession — we have written one of those already, and this is not it. This is about the class of failure where the ticket is the bug: where a maintainer, acting reasonably, reads a note, believes it, plans work around it, and only discovers on contact with the data that there was nothing to do. The work still costs what it would have cost. It just produces nothing.

The scoreboard

Five of those are phantom defects. The last one is different in kind and worth separating: the gap was real and the proposed fix was the defect. More on that below, because it is the one people are most likely to get wrong under time pressure.

The paragraph that listed what had been added

The cleanest case, and the one that generalises furthest.

An internal note read, in substance: "Added six real, common surnames that were missing: A (rank 42), B (rank 46), C (#56), D and E (#60–61), F (#66)."

That is a record of completed work. Read six weeks later by someone who was not there, it is indistinguishable from a to-do list. The word "missing" is right there in the sentence; the fact that it describes a condition that ceased to exist in the same sentence's action is carried entirely by the past tense of one verb.

So a work item was opened: these surnames are missing from the list. And the item cited the ranks — 42, 46, 56, 60 — as supporting evidence.

Those ranks were correct. They were correct because the entries were there, at those ranks, which is why the note could quote them in the first place. The evidence for the defect and the defect itself had been copied out of the same paragraph. Checking the file took under a minute and returned all five entries at ranks 42, 46, 55, 59 and 60 — matching the note to within the drift you would expect after the list had been deepened in the interim.

The lesson is not "write more clearly." The note was clear. The lesson is that a sentence describing a change is undated by default, and an undated change reads as a state. Added A, B, C and A, B, C are missing are the same sentence to a reader who does not know when it was written.

The event read as a state

Same shape, larger blast radius.

A note recorded that 33 archaic surnames with zero living bearers had been removed from a list, having been imported from an 11th-century text rather than from a population. True, dated, and correct.

Four days later a plan item said: remove 33 archaic surnames with zero living bearers.

Both sentences are true about our data. They are true about different versions of the same file, four days apart. And here is what made it worse rather than better: in the interim the file had been rebuilt from a registry, so the current version was a strict subset of a per-name registry with every weight equal to a registry count. There was not merely nothing to delete — there was nothing that could be deleted, structurally, because every row had a counter behind it.

The measurement took one script: 2,707 entries, all present in the registry under byte-exact comparison, zero weight discrepancies, coverage 99.98%.

The tempting cleanup at this point is to strike the old note as an error. That would be a second mistake. The old note is not wrong — it accurately describes an event, and we have published an article about that event which needs no correction. What the note lacked was a scope: which file, as of when. Two accurate statements about two versions of the same thing, sitting in two documents with no dates, present as a contradiction; and a contradiction in the documentation is a work item waiting to be written by the next person who notices it.

The mechanism that was never written down

A third item claimed a weight column blended three incompatible scales and needed to be reconciled onto one.

There had been three sources. There was one scale: at some earlier point they had been reconciled by fitting everything to a single coefficient, and the fitting step had never been documented. Recovering the coefficient from the data itself took an afternoon; once recovered, it reproduces 1,813 of 1,886 weights exactly, to the unit.

Then the interesting part. We measured what the proposed reconciliation would actually have changed:

And the cost side: those 73 entries were the only ones whose values came from an independent source. Rewriting them would have made the file fully self-consistent and permanently unverifiable — the only set against which the coefficient could ever be checked would have been overwritten by the coefficient.

A change with a measured benefit of zero and a non-zero cost is not neutral. It is a regression. It only looks neutral because "normalise the scales" is a sentence that sounds like hygiene.

The real gap with the wrong fix

The sixth item is the one that deserves care, because everything about it was reasonable except the conclusion.

Twenty locales had city lists with no population weights, so cities were picked uniformly — a small town as likely as the capital. That is a genuine defect, documented honestly, with sources on disk for some of the locales. The item said: load the weights we have.

Measured, the sources covered 563 of 810 cities across 14 locales — 69.5%. Not one locale reached 100%. The best was 90.4%, missing five cities.

The instinct is that 69.5% is better than 0%. It is not, and the reason is worth stating precisely: an unweighted list picks uniformly, which is wrong but unbiased across the entries. A partially weighted list makes the weighted entries deterministically more likely and the unweighted ones deterministically less likely — and the entries that lack a population figure are not a random sample. They are the ones with awkward names, contested boundaries, or ambiguous matches. So partial loading systematically suppresses exactly the entries that were hardest to resolve, and it does so while looking more precise than before.

The correct output was to load nothing and write down the 247 gaps as a manifest, so the next attempt starts from a list instead of from a rediscovery. "Some data is better than none" is false whenever the missingness is correlated with the thing you are measuring, which for geographic and name data it almost always is.

Why the cause is documentation and not data

Six items, six different countries, six different kinds of claim, one common origin: a sentence in our own notes that had been true when written.

This is structural, not a local failure of diligence. A measurement is dated by construction — you ran it, on a day, against a file, and if you re-run it you get today's answer. Prose has none of that. A paragraph does not know when it was written, does not know what it referred to, and does not degrade visibly. Documentation does not go stale loudly. It goes stale silently and then generates work.

We have a hard measurement of the rate. During the same revision, one count in our documentation — how many locales carried a particular kind of dataset — moved 7 → 9 → 10 within a single day as parallel work landed. Nine documents carried that count in thirteen places. And the report we wrote to fix all thirteen went stale twenty minutes after being written, because another job finished in the interim and made the number 10.

That is not an anecdote about irony. It is the constraint. Being careful is what produced the first version, so "be more careful" cannot be the remedy.

The remedy: a number without a recomputation is a landmine

The rule we now apply, and the only one from this exercise that survives contact with a real project:

Never write a count in prose without writing, next to it, how to recompute it.

A bare number states a fact about a moment. A number plus a recomputation states a fact about now. The difference costs one line and removes an entire failure class — because the next reader does not have to decide whether to trust the number; they can spend ten seconds finding out.

The second rule is about tense. Every factual sentence in project documentation should be classifiable, with no surrounding context, as exactly one of:

A sentence that cannot be classified is a phantom work item that has not been written yet.

Why "check first" is harder advice than it sounds

Everyone agrees with "verify the premise before planning the work." Almost nobody does it, and the reason is a genuine asymmetry rather than laziness.

Reading a document takes seconds and requires nothing. Running a check takes minutes, requires knowing what to run, and requires having the data at hand. So the natural order is: read, form a plan, then verify during execution. That order is exactly backwards — because by the time execution starts, the plan has been written down, scheduled, staffed and defended in a meeting. Verification during execution does not cancel work; it merely discovers, expensively, that the work was pointless.

There is also a specific reason phantom items feel solid: they are precise. "33 phantom surnames" carries a number, and numbers read as measured. That one was measured — four days earlier, against a file that no longer existed in that form. Specificity is not evidence of currency, and it is the single most reliable way a stale claim disguises itself as a fresh one.

Our own accounting for this revision: at least three full working sessions went to items that a one-minute scan of the data would have closed. The scan now exists and is re-runnable, which is the only durable output any of those sessions produced.

The checklist

1. Test the premise against the data as your first action — before planning, not during execution. If the premise cannot be tested cheaply, that is itself the first finding.
2. Never write a count without the means to recompute it. One number in our notes changed twice in a day and lived in thirteen places.
3. Date every factual sentence and fix its tense. "Added X" and "X is missing" are the same sentence without a date.
4. A record of what was added is not a list of what is missing. When a note enumerates items, check whether it is a receipt or a request.
5. Separate "the defect is not real" from "the fix is not correct." They arrive in the same shape and need different responses; one closes the item, the other replaces it.
6. Partial fixes can be worse than none when the missing part is not a random sample. Measure coverage before loading anything, and prefer an honest gap to a biased fill.
7. Fix the document in the same session as the phantom item. Otherwise the sentence that generated it stays in place and generates it again — we watched exactly that happen, twice, with the same paragraph.
8. Watch for one-sidedness. Every one of these defects existed in documentation and could never be reproduced against the data. A defect that only ever appears on one side of a comparison is usually a diagnosis problem, not a data problem.

What it actually cost, and what it bought

The three genuine items were real work and were done. The six were closed by measurement, and closing them took less time in total than implementing any single one of them would have.

The most valuable output of the revision was not a change to the data. It was the deletion and re-dating of the sentences that had produced the false items — because those sentences were still sitting there, still true-when-written, still ready to generate the same six items for the next person who read them carefully and acted in good faith.

Which is the uncomfortable general form. A database degrades slower than its documentation. The data is checked by validators every build; the prose around it is checked by nobody, ages continuously, and is the thing everyone reads first.

Data as of 2026-07-21

Every figure in this article was recomputed from the data on 21 July 2026, not taken from the notes under discussion — which would have been an unusually poor idea for this particular article. The revision described took place on 19–21 July 2026 across a name and address database covering 64 locales.

Where the figures come from:

• 2,707 entries, byte-exact registry match, 99.98% coverage — recomputed against the published per-name surname registry for the locale in question; that registry, its licence and its quirks are documented in Where Our Taiwan Name Data Comes From.
• Five surnames at ranks 42, 46, 55, 59, 60 — read directly from the shipped list, 298 entries.
• 1,813 of 1,886 weights reproduced by one coefficient; median rank shift 0; 73 of 2,222 entries affected — measured by re-deriving the coefficient from the data and replaying the proposed change without writing it.
• 60 of 60 city weights; 563 of 810 across 14 locales; 247 gaps — counted from the shipped city lists and the offline sources.
• 7 → 9 → 10 in one day, in 13 places across 9 documents — counted during the documentation revision itself.
• The given-name registry that was said not to exist is published by Istat, Italy's national statistical institute, covering 1999–2024.

The companion article, covering eight substantive errors of a different kind — wrong denominators, circular verification, zombie statistics — is What We Got Wrong About Name Data. The general method for testing this class of claim is in How to Audit a Name-Frequency Dataset.