A text file goes in, a film comes out

Last time I told you why I started this. I ended that post by saying I’d get into how it actually works. This is that post.

So what is MoVox, really? Here’s the shortest true answer I have: a text file goes in, a film comes out.

You hand it a plain .txt of a classical novel, the kind anyone can download for free in about ten seconds. What comes back is film: every scene painted in ink-wash, every character voiced, the pictures moving, subtitles burned in. No cloud, no subscription, nobody charging me per frame. The whole thing runs on one machine sitting on my desk.

Let me be precise about the word “film,” though, because it’s doing some quiet work in that sentence. MoVox works a chapter at a time. One chapter in, one MP4 out. A hundred and twenty chapters gets you a hundred and twenty files, and whether you want to call that one very long film or a hundred and twenty short ones is up to you. I think of it as a series.

And that’s not an accident of how I built it. The chapter is the unit because the chapter was always the unit. These novels were written to be delivered one 回 at a time, by a storyteller who stopped at the good part and told you to come back tomorrow. I’m keeping his pacing.

And while I’m being fussy about words, let me do the name too. I’ve been saying MoVox for two posts now without ever stopping to explain it.

It’s two halves. Mo (墨, mò) is ink in Chinese: the ink these books were printed with, and the ink the paintings are made of. Vox is Latin for voice. Ink and voice, which is the whole project in two syllables. Take the old written word and give it something to speak with. The domain fell out of it, movox.ink, and the 墨 sitting at the top of this page isn’t a font, it’s a Yuan dynasty calligrapher’s hand.

I’d like to tell you I agonized over it. Truthfully, naming things is the part of engineering I’m worst at. This one turned up early and never left.

That still hides all of the work, though. So let’s open it up.

First: why a crew, and not just one program.

I could have written this top to bottom as a single program. I didn’t. MoVox is a pipeline of agents, where each agent is one specialist that does exactly one job and hands its work to the next one down the line.

Let me define that word, because “agent” is pulling a lot of buzzword duty these days. When I say agent, I mean a piece of the system with its own job, own model, own instructions, and its own idea of what “done” looks like. It doesn’t know or care what the agent before it did, as long as what landed on its desk is what it was expecting.

Why build it that way? Two reasons, and both of them are boring engineering reasons.

First, these jobs have genuinely nothing to do with each other. Working out who is speaking a line of classical Chinese, and painting a mountain in ink-wash, are not the same skill. There’s no one model that’s best at both. Splitting the work lets me put the right model in each seat, and swap one out later when something better shows up, without disturbing anything else.

Second, and this is the one that saved me: things fail. A chapter takes hours to produce. If it dies at hour three, I don’t want to start again from zero. So every agent writes what it produced into a database, and every agent checks that database before it starts, skipping whatever is already done. You can kill the process in the middle of the night, start it again in the morning, and it picks up at the exact paragraph it stopped at. Boring, unglamorous, and it’s the difference between a project and a demo.

Alright. Five seats. Let’s walk down the line.

The Reader takes the raw novel and turns it into structure. It splits the file into chapters, then works through each chapter and breaks it into segments: this part is narration, this part is somebody speaking, and if somebody’s speaking, who, and in what emotional register.

I want to stop here longer than the others, because the Reader carries the whole pipeline on its back.

Here’s the thing to understand: nobody downstream ever reads the novel. Not one of them. The Actor doesn’t read the novel, it reads a row that says this sentence, spoken by this character, angry. The Painter doesn’t read the novel, it reads these characters, in this place, at this moment. The Animator and the Editor are further downstream still. They’re all working from the Reader’s notes, and none of them can tell the difference between a good note and a confidently wrong one.

So every mistake the Reader makes gets faithfully carried all the way to the screen by four agents who have no way of knowing anything is off. That’s why this seat gets the most of my attention, and why I check its output by hand more than anything else in the project.

And the real difficulty in it isn’t the part people expect. It’s names.

Take Romance of the Three Kingdoms. Liu Bei is 刘备. He is also 玄德, which is his courtesy name. To his own men he’s 主公, “my lord.” Others call him 皇叔, “imperial uncle,” or 使君, or a dozen other things depending on who is talking to him and how much respect they’re paying. Same man, and the book will use four of those names on a single page without ever pausing to tell you they’re the same person, because any reader in 1522 obviously knew that.

If the Reader doesn’t catch it, Liu Bei walks out of chapter one with five different voices. So there’s a whole pass that goes back over the finished novel, gathers up all the names, works out which ones are the same person, and merges them into one. Get that one step wrong and every step after it is wrong too, which is the failure I described above in its purest form: wrong speaker, so wrong voice, so the wrong man standing in the picture, and nothing downstream raises a hand to tell you. This one is getting its own post.

The Actor is the crew’s voice actor: every character gets a voice of their own, minted once and used forever.

And I do mean minted, not picked. I’m not choosing from a list of stock voices. I write a persona in plain words, something like an old general, gravel in his throat, never hurries, speaks from the chest, and the model generates a voice to match that description. Then it banks that voice across fifteen emotions, so the same character can be angry, or pleading, or arrogant, and still be recognizably himself.

There’s a small trick in there I’m fond of. The randomness is seeded from the character’s name plus the text of the persona, so the same description always produces the same voice, every run, forever. Change one word of the description and you deliberately get a different person. That’s not a limitation I worked around, it’s a dial I wanted.

And the narrator is going to be me. Reading these books to my kids in my own voice was the point I started from.

The Painter composes an ink-wash keyframe for every scene. That’s the picture you’ll be looking at while that stretch of the chapter is read to you.

Making a beautiful ink-wash painting is, honestly, the easy part now. Models are good at that. The hard part is that the Monkey King has to be the same monkey in chapter two and in chapter eighty. Ask a model to paint the same character twice and you’ll get two cousins. So each major character first gets a character sheet, a set of reference images of that one face, and from those I train a small adapter for the model. That’s a LoRA, if you’ve seen the term thrown around: not a whole new model, just a light attachment that teaches an existing model one specific thing, in this case one specific face. It’s cheap to train and you can stack several of them, which matters when three characters are standing in the same shot.

The Animator takes each still keyframe and gives it a few seconds of motion. I want to be clear about scope: this is not a cartoon. Nobody’s lips are moving. Water moves, cloth moves, smoke drifts, the camera breathes. Eight to ten seconds of a painting being alive. That’s the target, and I think it’s the right one for these stories.

The Editor does the assembly. It stretches each clip to match how long the narration for that scene actually runs, upscales everything to 1080p, lays the subtitles on, and stitches the chapter together into a single file. One chapter in, one MP4 out.

The subtitles turned out to be a much bigger job than “put the text on the screen.” Getting a line of text to appear and disappear exactly with the voice saying it reaches backwards up the whole pipeline: it changed how the audio gets generated in the first place, and each audio file now ships with a small companion text file next to it recording what was actually said and how it was broken up, purely so the Editor can line the two up later. That’s another post.

Now, where do I sit in all of this?

Between all of them. That’s the part I care most about and the part that’s easiest to leave out of a diagram.

There are checkpoints down the whole line, and I sit at them. I built myself a little web viewer over the database so I can read what the Reader produced and see the lines it got wrong. Character sheets don’t go anywhere until I’ve looked at the faces and approved them. Audio gets checked. Nothing reaches the finished film without a person having looked at it and said yes.

I wrote a whole post about why that matters, so I won’t relitigate it here. Short version: the model has no taste. I do. Not slop, craft.

Now the honest part, because I’d rather tell you than let a nice diagram imply otherwise.

Not all five seats are filled yet.

The Reader works, on a full-length novel: Romance of the Three Kingdoms, all hundred and twenty chapters of it. The name-merging works. The persona writing works. The Actor works: characters get minted voices, and chapters get spoken out loud. That half of the line is real, it’s running, and I’ve listened to the output.

The Painter and the Animator are designed and wired into the pipeline, and they are still stubs. A stub means the room is built, the door is hung, the wiring is in the walls, and there’s no furniture in it yet. The pipeline runs straight through them today without producing anything.

The Editor is the odd one out, and it’s the good kind of odd. It partly works, and it got there sideways. I originally wrote it to help me debug the audio: I needed to actually watch a chapter play through to hear where the voices went wrong, rather than clicking through hundreds of separate audio files. But building that meant solving the subtitle timing for real, and the timing is now solid. So what it produces today is a genuine, watchable chapter file: correct audio, correct subtitles, correctly timed, playing over a gray rectangle where the painting is going to go.

That gray rectangle is a fair picture of where the whole project stands. As of this writing, MoVox reads, speaks, and cuts. It doesn’t see yet. That’s what’s next, and building in public means saying where the line actually is instead of where I’d like it to be.

Three constraints shaped every decision above, and each one is getting its own post, so I’ll only plant the flags here.

It all runs locally, on my own hardware, no API calls to anybody. Every model in the pipeline is under a permissive open license, Apache 2.0 or MIT, and my reason for that is not idealism, it’s a practical one I’ll explain when I get there. And all of it has to fit on a single consumer graphics card, which turns out to constrain the design far more than you’d guess.

Those three are most of the interesting engineering in this project. 工欲善其事，必先利其器 — if you want to do the work well, first sharpen your tools. A lot of this project has been sharpening tools.

That’s the whole pipeline. A text file goes in, and a film comes out one chapter at a time, made by five specialists who never get tired and one guy who checks their work.

And at the far end of it there’s still a kid who can’t read the original, getting the whole adventure anyway.