Skip to content

The problem

A studio where you paste a script, give it a short recording of a voice, and get back a polished narration in that voice: Hindi first, English too. It had to run on the hardware I had: a gaming laptop with an NVIDIA RTX 3050 (6 GB).

The engine is Chatterbox Multilingual by Resemble AI, an open model that clones a voice from a few seconds of audio (zero-shot). Around it I built a FastAPI backend with a job queue, long-form chunking up to 30,000 characters, resumable jobs, a React studio UI and a test suite.

The voice was recognisable. But played to anyone who knew the speaker, it gave itself away within seconds. Similarity scores looked fine; listening didn’t.

It sounded spliced

Every few seconds the background hiss vanished and came back: tiny holes of perfect silence between chunks.

It sounded flat

The delivery was even and careful, like someone reading, not speaking.

The pace was wrong

The real speaker talks fast and dense; the clone was slower and over-articulated.

Odd glitches

Occasionally a chunk mumbled, repeated itself, or ended with the voice dropping mid-sentence.

Instead of swapping models, I asked: what in my own pipeline makes the output recognisably synthetic? That meant reading the model’s source code, measuring the reference audio, and testing every stage between “text in” and “MP3 out”.

12
Weak points found
across reference, settings, stitching, speed
5
Improvement phases
each measured before moving on
80
Automated tests passing
with no files left behind by the suite

Next: how the pipeline works, then what I fixed.