Sixteen thousand characters gone

In the last note I said I went quiet to make a demo, and then found out a lot of “the picture is wrong” was really “we asked for the wrong picture.” This one is the same shape of mistake, one seat upstream.

The Reader is the agent that takes the novel and turns it into structure: this is narration, this is dialogue, this is who’s speaking. I already told you why that seat carries the pipeline. Nobody downstream reads the book. The Actor reads a row. The Painter reads a row. If the Reader drops a sentence, that sentence is gone from the film. If it hands the line to the wrong man, the wrong man says it, in the wrong voice, with the wrong face on screen, and nothing later in the line has any way to know.

In July I told you the Reader works, on all 120 chapters of Romance of the Three Kingdoms. I believed it, and I had a reason to. What I didn’t tell you is how it got there.

Not “most of the book.” The book.

Back in June, a few weeks before that post, I concatenated every segment the Reader had written, chapter by chapter, and diffed it against the source file on disk. Not a vibe check. A character count of the actual Chinese.

15,753 characters, cleanly missing, across 104 of 120 chapters.

That’s not a typo in a name. That’s whole stretches of the novel. Chen Lin’s war proclamation in chapter 22, gone, the entire document. The fog in chapter 46, where Zhuge Liang borrows the arrows, full of holes. A protest from Ding Yuan in chapter 3, sitting in the gap between two segments that had closed ranks as if he’d never spoken.

The part that still bothers me is that I had a checker for this. There was a reviewer whose job was to look at each block of text, see what the model emitted, and say whether the source was actually covered. It was sitting in the repo. It was never called. The function existed, and the pipeline walked right past it, took whatever the model returned, and wrote it down.

The model drops content on successful calls. Of course it does. You can’t see that from a nice clean JSON array. You can only see it if something compares the array to the page.

So I wired the checker in, with retries, and a floor that splices anything still missing back in as narration. Coverage went from about 89% to 99.96%. That’s the Reader I told you about in July. And that’s where I made the next mistake: I treated 99.96% as done.

A coverage percentage is not coverage.

You can add up the Chinese characters in the segments, add up the Chinese characters in the source, and get a number that looks great. That number doesn’t care if the text is in the right order. It doesn’t care if a line was emitted twice. It doesn’t care if the model “helpfully” rewrote a sentence and the characters still roughly match.

We’re reading a book aloud. A dropped character is a defect an ordinary listener can catch. A duplicated line is the same defect in the other direction: you hear the book stutter. When I finally looked for them, there were 24 lines in the novel that were going to be spoken twice. There were also 3 lines the model had re-said under a speaker cue it invented, words the book never put in that man’s mouth. And a rewritten sentence is a different book.

So the real test isn’t a net count. It’s a forward walk: does this segment appear in the source, here, as a substring, in order, without having already been used? If it doesn’t, something was invented, reordered, or lifted from somewhere else on the page.

I’m labouring this because I’ve watched people, including me, comfort themselves with a percentage. I did it on the pictures too. The frame reviewer’s totals looked fine while it caught fewer than half of the defects I could see with my own eyes. Same shape, different seat.

I stopped asking the model to be complete.

That sounds like giving up. It’s the opposite.

A local model will never give you 100% coverage on a chapter of classical Chinese. Not reliably, not on a good day, not if you write a sterner prompt. I accept that. The product requirement doesn’t. Every character of the source has to reach the audio and the subtitles, for every novel, and that’s not negotiable. You can’t trade a prettier split or a better speaker guess against a missing sentence. The order is fixed: the words first, then who said them, then where the pause sits.

So after the model has had its tries, a deterministic pass diffs the segments against the chapter file on disk and forces anything still missing back in. It doesn’t rewrite the missing span to sound nicer. It attaches it to a neighbour, or splits a segment so the gap has a place to live. If it can’t keep both the text and the speaker, it keeps the text and marks the row for me to look at.

There’s a small, almost comic version of this I want to mention, because it’s the kind of bug you only find once you’ve decided 99.96% isn’t the target. The first floor wouldn’t splice back a run shorter than 4 characters. The leftover was 167 characters in 76 little crumbs across 38 chapters, almost all of them structural: 诗曰 (“a poem goes”) before a poem, 却说 (“now, as for…”) at the start of a section, half a name on one side of a cut and the rest of the name on the other. Each crumb was short enough to ignore on a chart. Each one is a word a child would hear missing. The chapter-level pass now merges those too.

Romance of the Three Kingdoms, the copy I’m actually building against, as of early August: 120 of 120 chapters exact. Zero missing. Zero duplicates. That’s the number I’ll sign.

And then a better model rewrote the book.

One more, because it’s the same lesson with a twist. Later in August I tried a much bigger model as the Reader, on the second computer I’ll get to in the last note. It segmented better. It also, quietly, copy-edited the novel. It added a closing quote mark the source doesn’t have. It expanded 曰 (“said”) into 曹操曰 (“Cao Cao said”) in the displayed text. It fixed what looked like a typo in the source, changing one character of a name to the character it “should” be. Every one of those edits is helpful. Every one of them is a fidelity violation, and a retry doesn’t clear it, because the model makes the same edit every time.

A bigger, smarter model is more likely to do this, not less, because being helpful is what it was trained for. The fix, again, wasn’t a prompt. It’s another deterministic pass that lines each segment up against the source and hands it back the exact span of the book it claimed.

What this is really about, if you’re not building a classical-novel pipeline.

It’s about not confusing a model’s confidence with a contract.

The Reader is an agent. It has a job, a model, a notion of done. Left alone, “done” means “I produced a plausible JSON array.” That’s a different sentence from “this array is the chapter.” The second sentence has to be checked by something that can’t be talked out of it. In my case that something is a diff against the file on disk. In yours it might be a schema, a test, a round-trip, a hash. The tool changes. The move doesn’t.

The expensive part is how long I believed the first nice number. I’m glad I didn’t stop at it, because every picture I make from a missing sentence is a picture of a book I didn’t actually put in.

The next note is about the other half of “reads and speaks”: trying to make the narrator sound like me, and finding the limit of what a small pile of recordings can teach a model.