Six minutes is not a voice

If you’ve been reading these in order, I went quiet to paint, and then the book turned out not to be quite the book. This one is the other half of what I claimed in July was already working: the voices. The characters have them. The chapters get spoken. The question left was whether the man reading the book would be me.

I said that out loud in July. Reading these novels to my kids in my own voice is why the project exists. A cast of invented generals can sound like invented generals. The narrator is supposed to be the person who sat on the edge of the bed.

So I tried to fine-tune my way there. Then I measured what I’d actually bought.

The punchline first, because the rest of this is how I found it.

Zero-shot off a real recording of me already sounds like me. That’s the path that ships today. I read a few sentences into a microphone, that clip becomes the reference, and the model speaks the chapter in that voice. No extra training. I recognised myself immediately.

What had been making the narrator not sound like me, for a while, wasn’t a missing fine-tune. It was a fake reference: a synthetic clip the voice-design model had generated, at 16 kHz, which is below what the speech model even outputs. Swap in a recording of my actual voice and the problem mostly went away. I’d been getting ready to train my way out of a clip that was never me.

I trained anyway, because “mostly” isn’t the same as “the way I talk,” and because I wanted to know where the ceiling was.

Six minutes of isolated sentences will teach a model your timbre. They won’t teach it how you talk.

The pile was 45 short takes, 6.2 minutes in total. I sat in a recording page I built for this, read one sentence at a time, saved the take, went to the next. It’s a miserable way to read a book, and I now think it’s also a miserable way to build a training set for someone who’s supposed to narrate one.

I trained three adapters. I won’t walk you through the hyperparameters. The useful part is the pattern that survived all three.

The adapter reliably learns who’s speaking. When it stays at a normal speed and a normal volume, it does sound like me, mostly. Then, inside a single generated stretch, the level and the pitch start climbing. Phrase by phrase it gets louder and higher, like a man who’s decided the story just got exciting. At the next stretch it snaps back down and starts climbing again. If the drift goes far enough it leaves my range entirely and sounds like a different person.

The base model, the one with no adapter, does the opposite. Its voice falls a little across a phrase, which is how real speech works. The fine-tune had taught the model that loudness was a free parameter, because my 45 takes were recorded at levels nearly 16 dB apart. So I normalised the training audio. That fixed some clips outright and did almost nothing for others, and the pitch jumps barely moved.

I also learned, the expensive way, that you can’t judge this on a cute eight-sentence probe. A probe predicted twice the improvement a real chapter then delivered. The only test that counts is generating a chapter and listening to it, which is slow, which is why I kept cheating with probes, which is why I kept being wrong.

There’s a second problem hiding under the first. Isolated sentences produce what’s called citation prosody: reset the pitch, stress every word, land a clean fall at the period, every time. That’s how you read a flashcard. It’s not how you tell a child a story, and it’s not how a chapter sounds. So even a perfectly level, perfectly identified 6.2 minutes would still have been teaching the wrong register. The corpus wasn’t only too small. It was the wrong shape.

I stopped turning knobs.

This is the part I want to be stubborn about, because the temptation is always one more run.

Rank, learning rate, which parts of the model the adapter attaches to, how hard you turn it up at inference. I have numbers for all of those, and they’re written down. They’re not the binding constraint. Six minutes is. Another week of training against this pile would have produced another adapter that sounds like me until it gets excited. I have that adapter. I don’t need a fourth.

So I built the thing that would let me grow the pile, and then I didn’t grow the pile.

The studio can now hand me a long passage of the novel’s own narration, a minute or so of reading, and I read it in one take, pausing between sentences. Software cuts the take back into per-sentence training pairs. That cutting is forced alignment, not transcription: I already know the words and the order, I just need to find where each sentence landed in the recording. I tested the cutter by gluing my old 45 clips into one fake long take with known boundaries, so I wouldn’t have to trust a demo on data I couldn’t check. Every cut the system accepted landed within 306 ms of the true boundary, median 13 ms. The ones it wasn’t sure about, it threw out rather than guess.

What’s left is me, sitting down, reading about 32 minutes of passages into a microphone.

I haven’t done it. I paused this in the middle of August to work on the pictures, and the pictures still aren’t done. The narrator you’d hear in a demo today is the zero-shot one, which already sounds like me, which is why this pause is honest and not an emergency. The adapter is optional and off by default. Fine-tuning was never what stood between my kids and my voice. A real clip was. A bigger corpus is what stands between “sounds like me” and “talks like me.”

The lesson I’m keeping, for the next time I want to train my way out of a feeling.

Measure the ceiling before you decorate it. If the defect is baked into the data, more of the same data scales the defect right along with the signal. If the data is the wrong shape, a better optimiser just teaches the wrong shape more efficiently. And if the next hour of progress is a human task, no amount of pipeline work replaces it. I can build a recording studio. I can’t build the recording.

Chapter one already plays in my own voice, the zero-shot one, and I listen to it while I argue with pictures. (The other 119 chapters were rendered back when the narrator was still the synthetic clip, and they’ll need a re-render.) That part of July was true. It just wasn’t finished in the way starting a fine-tune implied.

The last note in this set is a correction. I told you this all ran on one machine on my desk. That sentence isn’t true anymore, and the second machine didn’t do what I hoped it would.