Fine-tuning a dictation cleanup model: designing the corpus, and the category 25% self-correction data still missed.

The one-paragraph version

When you dictate, you don't speak in clean sentences. You say "um, so, can you send the — wait, no, email the report to Priya by, uh, Friday." Something has to turn that into "Can you email the report to Priya by Friday?" I could have written a long instruction telling a general-purpose AI how to do that, and at first I did. It was slow, and it kept wandering off the job — answering the question instead of tidying it up. So I trained a small AI to do only this. That needs examples: tens of thousands of messy in, clean out pairs. Here's the catch — the one thing I refuse to collect is real dictation. The whole point of the app is that your voice never leaves your Mac. So I had to manufacture the examples, which turns into a design question: what does "cover the range of things people actually dictate" even mean? I ended up building a menu system — 29 topics, how formal it is, what shape the sentence takes, how long, how messy — and generating an equal number of examples for every combination. Then a trick that matters more than it sounds: write the clean sentence first, then mess it up on purpose, so the right answer is known in advance instead of guessed afterwards. And then the interesting failure: a quarter of my entire dataset was people correcting themselves mid-speech, and the model still got a whole class of corrections wrong — the ones that cross from one sentence into the next.

What the words mean

• Fine-tuning. Taking a general-purpose AI and training it further on one narrow job until it stops needing to be told what the job is. The instructions move out of the prompt and into the model itself.
• LoRA. A cheap way to fine-tune. Instead of adjusting all two billion numbers in the model, you train a small patch that sits alongside them — small enough to train on a home machine, and you can merge it in afterwards.
• Synthetic data. Examples you generate rather than collect. Necessary here, because collecting the real thing would mean recording people's dictation — exactly what the app promises never to do.
• A self-correction. Changing your mind mid-speech: "send it Tuesday — actually, make that Thursday." The cleaner has to keep only the final answer, never the cancelled one.

What I found, plainly

• A longer instruction wasn't the answer. Explaining the job to a general model took about 1,100 words of setup before it even saw your sentence, every single time — and it still broke character. Training the job into a smaller model cut that setup by roughly three-quarters and made it behave far more predictably.
• Covering the space had to be deliberate. Generate examples freely and you get a thousand variations of "set a timer" and nothing resembling a rambling voice memo. I built a grid instead and filled every cell equally: every topic, at every formality, in every shape, at every length, at every level of messiness.
• Not every combination is real. A formal shopping list isn't a thing anyone says. Pruning the impossible combinations cut the grid from 522 down to 119 sensible ones — which is why the data isn't full of nonsense nobody would ever dictate.
• Write the clean version first. If you ask an AI to produce messy speech and then clean it, it quietly rewrites things and you never learn the true answer. Generating the clean sentence first and then deliberately roughing it up means the correct answer is known by construction.
• The failure is the interesting part. Self-corrections were the single biggest category in my data — about a quarter of everything. The model still failed on corrections that span two sentences, because every example I'd built kept the correction inside one sentence. Having lots of data about a topic is not the same as having data about every shape that topic takes.
• Then my own app threw away the fix. Once the model produced the right answer, a safety check rejected it for being "too short" — because correctly deleting a cancelled clause makes the output shorter than the check expected. The wrong answer was long enough to pass.

Why this matters if you don't build these

Two things worth taking away. First: "we have lots of data about X" is not the same as "we have data about every shape X comes in." A quarter of my dataset was the exact phenomenon that later failed. The gap wasn't quantity — it was a distinction I hadn't thought to make. Second: an upgrade isn't an improvement until the whole system agrees. A better model plugged into old safety checks can be silently cancelled out by them, and everything will still look like it's working.

Want the receipts — the taxonomy, the generation method, and the exact numbers? Flip the switch above to Full technical.

Why not just write a longer prompt

Cleanup started as a general instruction-tuned model — Qwen3-4B-4bit — steered at runtime by a static prefix of roughly 1,100 tokens: system rules plus ten few-shot examples. It worked. It was also slow, it burned prefill on every request, and the model kept finding new ways to break character.

The evidence for that last point is the guard stack that accumulated in acceptOutput. Each guard is a fossil of a way a general model went off-task: answering the question in the text instead of cleaning it, emitting markdown headers, adding unsolicited bullets, substituting short phrases, wrapping output in quotes, leaking <think> blocks. Every one is a rule I had to add because a model that knows how to do everything will occasionally decide to do something else.

A longer prompt is the obvious fix and the wrong one. Prompt length is paid on every single dictation, and the failure modes it patches are a symptom: the model has no strong prior that this is a narrow rewriting task. The bet is to move the task out of the prompt and into the weights. The prompt drops to ~256 tokens, and the failure surface narrows from "might do anything" to "might mis-handle a specific correction shape."

Why LoRA

A full fine-tune of even a 2B model means optimizer state for every parameter — impractical to do repeatedly on a Mac, and I wanted to iterate on the corpus, not schedule cloud runs.

LoRA trains a small low-rank delta beside the frozen base weights; QLoRA keeps that base quantized during training. Rank 16 on mlx-community/Qwen3.5-2B-MLX-4bit trains locally in hours, and the artifact is an adapter you can fuse into standalone weights once it's good enough. The iteration loop is the whole point: the corpus went through three major revisions, and each one had to be cheap enough that revising it was a normal Tuesday rather than a budget decision.

The fuse has one trap worth stating, because it cost a release. A plain 4-bit fuse rounds the low-rank delta away. It collapsed the model from 100% to 57.8% held-out self-correction accuracy while validation loss and filler removal both still looked perfect. Any re-fuse has to go --dequantize → bf16 → re-quantize to 8-bit.

The data problem

Supervised fine-tuning needs raw → clean pairs — tens of thousands of them. There is no public corpus of real dictation with cleaned targets, and the one source that would be ideal is the one I will never touch: actual user dictation. Pomvox's entire pitch is that your voice and transcripts never leave your machine. A corpus built from user speech would contradict the product at its foundation.

So the corpus is fully synthetic — no user dictation, no scraped speech. Which converts a data-collection problem into a design problem, and the design problem is the interesting one:

What does it even mean to cover the space of things people dictate?

Generate freely and you get a thousand variations of "set a timer for ten minutes" and nothing resembling a rambling voice memo about a meeting. Free generation collapses toward the mode. Coverage has to be imposed, not hoped for.

The taxonomy: five menus

Every example is one pick from each of five axes:

The 29 categories sit in five domains reflecting where dictation actually gets used: interpersonal messaging (friends/family, colleagues, group chat, personal and work email, quick acknowledgements), productivity/self (notes to self, todos, shopping lists, reminders, scheduling, journal, brainstorming), assistant/command (assistant commands, search, navigation, coding queries, tech how-to, translation), content creation (social posts, reviews, document dictation, meeting notes), and life admin (customer support, appointment booking, health log, finance, recipes, travel).

Not every combination is legitimate, and that pruning matters. A formal shopping list isn't a thing anyone says; a formal reaction barely is. Each category declares which formality levels and utterance types are plausible for it, cutting the grid from a naive 29 × 3 × 6 = 522 down to 119 real combinations. Without that step, a meaningful slice of the corpus would be teaching the model to imitate speech nobody produces.

The last two axes — length and disfluency — are flavors: they shape how something is said, independently of what is said. They cross into a 3×3 grid over every combination:

119 (category × formality × type) × 9 (length × disfluency) = 1,071 cells
1,071 cells × 25 pairs each                                 = 26,775 pairs

Every cell gets exactly 25 pairs. Coverage is even by construction rather than by sampling luck — perfectly balanced across length and across disfluency, with domain totals falling out of how many categories each domain holds.

The clean disfluency level is worth calling out: those rows have raw == clean. They are no-op pairs, and they exist to teach the model not to edit text that is already fine. Without them, a cleanup model learns that its job is always to change something, and it will happily "fix" a sentence that was already correct.

Clean-anchored generation

The obvious way to build these pairs is to ask a model for a messy utterance and then ask it to clean it. That approach is broken, and it breaks in a way that is hard to detect: the model drifts. It paraphrases, under-cleans, or quietly invents a different sentence — and now your ground truth is whatever it produced second, which you have no independent way to check.

So generation runs in two stages, anchored on the clean side:

1. Generate the clean utterance for the chosen grid cell. Normalize it — capitalization, spelling, question marks on questions. This is the target, and it is now fixed.
2. Re-speak that exact sentence with fillers, false starts and self-corrections at the requested disfluency level. This is the input.

Because the clean side is the anchor and the raw side is a messier rendering of that exact sentence, removing the disfluencies from raw recovers clean by construction. The pair is faithful because of how it was built, not because a model was trusted to be faithful. (For clean-level rows, stage 2 is skipped and raw = clean.)

Every row then passes slot, preservation, no-content-loss/no-invention, production-guard and format gates at generation time, and violations are discarded, never repaired. That rule is load-bearing. A repaired row is a row whose generator was wrong, and patching it keeps the bad generation in your data wearing a clean shirt — it teaches the wrong thing quietly, and it suppresses the signal that your generator needs fixing.

What the corpus actually became

That structural grid was the first design. The v2 regeneration reorganized the corpus around behavioral classes — what cleanup skill a row exercises — rather than only around subject matter. That corpus is 53,500 rows, and the class mix is the important table:

Sources are mixed deliberately: 29,425 templated, 14,448 from a local Qwen3.5-27B-4bit, 9,627 from other local MLX generation. All 53,500 rows passed the guards, zero errors.

Note the top row, because the rest of this post depends on it. Self-correction is the single largest behavioral class — 13,375 rows, a full 25% of the corpus.

The category 25% self-correction data still missed

v2 shipped. Then five failures came back from real use, all the same shape:

raw       Let's meet Thursday. No, no, wait, uh we'll meet Friday actually.
expected  Let's meet Friday.
v2 got    Let's meet Thursday.

It pasted the day you cancelled. Note the failure mode: the output is fluent, clean, and plausible. Nothing looks broken. There's no error to notice — you just send the wrong day.

Here is what makes this worth writing about. A quarter of the corpus was self-corrections. If you had asked me whether the model had enough self-correction data, I would have shown you that table and said obviously yes. The gap wasn't quantity. It was a distinction the taxonomy didn't encode.

Every self-correction row I had generated kept the correction inside a single sentence — "tuesday wait no friday at noon". Not one crossed a sentence boundary. My axes were category, formality, type, length, disfluency and behavioral class; "where does the correction land relative to a sentence boundary" was not an axis. So the generator never varied it, and the eval set — built from the same taxonomy — could not test it. Training and eval shared a blind spot because they shared a schema.

And the eval said everything was fine. v2 scored 24/28 overall while scoring 0/5 on a category its eval set did not contain. An aggregate is the easiest place in the world to hide a total failure: five zeroes disappear into twenty-three ones.

The out-of-distribution number was the only honest signal, and I misread it for a full release cycle. On Disfl-QA — real human disfluency, Wikipedia-QA register, collected by someone else, deliberately never trained on — v2 scored 9.0%. I read that as "different register, of course it's low." It was not a register mismatch. It was a missing category, visible the entire time.

Digging in surfaced a second absence, worse than the first: corrections that replace the interrogative phrase itself ("Where are… or what city has…") are roughly 66% of real human disfluency and were 0% of the corpus. The majority shape of the phenomenon, entirely absent from a corpus where that phenomenon was already the largest class.

The fix was 3,900 additive rows covering cross-sentence and interrogative-phrase corrections, de-leaked to 52,239 training rows. The prompt was held byte-frozen across versions on purpose, so the corpus was the only variable and the resulting numbers actually attribute to it:

eval_v3 is now scored and gated per category, never as an aggregate — on slot accuracy, preservation and guard behaviour separately.

The guard that threw away the fix

With v3 producing the correct output, the app's own acceptOutput rejected it and pasted the raw disfluent text instead.

acceptOutput had a flat length floor: reject if len(out) < 0.30 × len(raw). That encodes an assumption — cleanup only ever trims filler — which was true for every model before this one. A self-correction violates it directly, because resolving one legitimately deletes an entire clause.

The floor was inverted on precisely the utterances the release existed to fix: it admitted the wrong answer and rejected the right one. Shipping the new model id alone would have converted "pastes the wrong day" into "pastes your raw mumbling" — a worse product, from a strictly better model.

The fix drops the floor to 0.15 when the raw carries a correction marker (wait, scratch that, i mean, make that, or rather, actually, hold on, let's say, no, no). A bare no is deliberately excluded — "There's no rush." is ordinary content. Note what the fix does not do: 0.308 still clears 0.15, so the wrong answer is still admitted. The guard was never the thing that could tell right from wrong. The model fixed the semantics; the guard just got out of the way.

Odds and ends worth knowing

Don't select checkpoints on loss. Validation loss flatlines early while behaviour stays non-monotonic — the model keeps changing which utterances it gets right long after the curve stops moving. All 30 surviving checkpoints were scored behaviourally; iter 14500 won on the out-of-distribution score.

Training died at 77% — iter 15,000 of a planned 19,590 — to a Metal InnocentVictim GPU reset. Not resumed, deliberately: --resume-adapter-file restores weights but not the LR schedule, so resuming would restart warmup at peak LR on an already-converged model. The selected checkpoint predates the crash by 3,000 iterations, so the truncation is cosmetic.

Shipped with a known miss. reg_003 — a chained triple correction in the old lowercase, unpunctuated shape — regressed to 43/44 against v2's 44/44. It is anti-correlated with the cross-sentence gate across the entire run: it passed at 7 of 30 checkpoints, nearly all before iter 5000, and never once alongside cross-sentence ≥121/122. One regressed row against 19.5% → 100% cross-sentence is a trade worth making, and I'd rather state it than average it away. (It has since passed on device twice during release verification, so the published regression may be narrower than the card claims. I haven't re-measured properly, so the card's number stands.)

What I'd tell you to take from this

1. Impose coverage; don't hope for it. Free generation collapses toward the mode. A grid with equal fill per cell is boring, and it works.
2. Prune the impossible cells. 522 → 119. Combinatorial completeness isn't the goal; plausible completeness is.
3. Anchor on the answer you already know. Generate the clean target first, then corrupt it. Faithful by construction beats faithful by inspection.
4. Discard, never repair. A repaired row hides a broken generator.
5. "Lots of data about X" ≠ "data about every shape of X." 25% self-correction data, and a whole sub-type missing. Ask what distinction your schema fails to encode — that's where the hole is.
6. Your eval inherits your training set's blind spots when both come from one taxonomy. Only outside data breaks the symmetry. Take your worst OOD number seriously; mine was visible for a release cycle and I explained it away.
7. Never gate on an aggregate. 24/28 concealed a 0/5.
8. Re-derive your guards when the output distribution changes.

The model is public, with the full per-category tables, the five verbatim production cases, and the known gaps. The app that runs it is MIT-licensed.