Skip to content

What I fixed

Every fix below started with evidence: a line in the model’s source code, a measurement of the reference audio, or an A/B test on the GPU. Numbers marked client voice were measured on the real client reference (not published here for consent reasons; see Responsible use).

2,930 ms130 ms
Digital silence
7-chunk Hindi narration, client voice
0.9330.944
Speaker similarity
same text, same seeds
3 runs1
FFmpeg passes per upload
and zero temp files
01
Reference

Only the first 6–10 seconds of the reference were used

Problem
The model takes speaking style from the first 6 s and timbre/room sound from the first 10 s of the reference. Whatever happened to be at the start of a 27.6 s file decided the clone, including distorted parts.
Evidence
Found in the model source: ENC_COND_LEN = 6 s and DEC_COND_LEN = 10 s; only the speaker embedding sees the whole file.
Fix
Score every 10 s window (speech density, clipping, quiet edges) and rotate the best one to the front, so the rest still feeds the speaker embedding.
Result
For the client voice the 8.3–18.3 s window was chosen automatically; the clip is built in memory and cached once per voice.
02
Reference

The reference was clipped and noisy, and the model copied both

Problem
The decoder faithfully reproduces distortion and hiss from the reference.
Evidence
Measured: peak at 0 dBFS, 235 hard-clipped samples, constant background noise around −40 dB.
Fix
Repair clipping, remove low rumble (<70 Hz), apply only mild denoising (heavy denoise sounds robotic) and peak at −1 dBFS.
Result
285 clipped samples repaired before conditioning; uploads are converted once, straight to 24 kHz mono.
03
Settings

Expressiveness was set below the model’s default

Problem
exaggeration = 0.3 (default 0.5) produced a flat, careful ‘reading’ voice; cfg_weight = 0.7 forced slow, over-articulated pacing on a fast speaker. Values were hard-coded in two places with hidden English overrides.
Fix
Moved to configuration (0.5 / 0.4 / 0.75), removed the overrides, and tuned by measurement across 17 GPU runs.
Result
Stable, more natural delivery; a sweep showed 0.65+ starts to babble on this voice, so the safe range is documented.
04
Settings

Bad chunks went straight into the final file

Problem
Each chunk was sampled once. Mumbles, repeats, cut-off endings or 40 s babbles were never caught.
Fix
A GPU-free check per chunk: speech rate vs text length, the 40 s generation cap, and dead air > 1.2 s. On failure: one retry with a new seed, keep the better take.
Result
Normal chunks cost nothing extra; failures are caught. In the sweep, every real babble was flagged.

Stitching chunks together Phase 3

Section titled “Stitching chunks together ”
05
Joining

Gaps between chunks were perfect digital silence

Problem
Chunks were concatenated with generated zero-silence. The voice carries room noise at about −45 dB, so every few seconds the background vanished and came back: the classic splice sound.
Evidence
2,930 ms of exact digital silence in a 69 s narration.
Fix
Fill pauses with the recording’s own room tone, overlap-added from the quiet moments between words, with equal-power fades at every boundary.
Result
Digital silence fell from 2,930 ms to 130 ms; the noise floor is continuous.
06
Joining

Pauses were uneven and mechanical

Problem
Each chunk kept its own leading/trailing silence and then a fixed 250 ms or 600 ms was added on top.
Fix
Trim each chunk to its speech (+40/60 ms), then size pauses by punctuation: comma ≈160 ms, sentence 350 ms, question ≈400 ms, paragraph 750 ms, with ±15% deterministic variation.
Result
Rhythm follows the text instead of the chunk boundaries.
07
Joining

The voice dropped at mid-sentence cuts

Problem
When a long sentence had to be split, the model appended a full stop, so the voice fell as if the sentence had ended.
Fix
Mid-sentence chunks get a trailing comma, keeping a continuing intonation.
Result
Long sentences now flow across chunk boundaries.
08
Joining

Volume jumped from chunk to chunk

Problem
Chunks were not level-matched before joining.
Fix
Each chunk’s speech level is matched to the median (within ±6 dB, peak-safe).
Result
Even loudness across the whole narration.

Polish and speed Phases 4–5

Section titled “Polish and speed ”
09
Loudness

Loudness normalisation pumped the background

Problem
Single-pass loudnorm rides the gain like an automatic volume control, lifting the noise in every pause.
Fix
Measure once, then apply one fixed gain (linear mode) to −16 LUFS, in the same encode that writes the AI-attribution tags.
Result
Natural dynamics, and one file rewrite instead of two.
10
Speed

GPU memory was released after every chunk

Problem
torch.cuda.empty_cache() after each chunk forced the allocator to rebuild its memory pool every time.
Fix
Release once per job; the model stays resident and an idle janitor unloads it after 5 minutes.
Result
Steady ≈1.3× real-time once warm.
11
Efficiency

The same work was repeated on every chunk

Problem
The reference file was re-read and re-hashed for every chunk; uploads took up to three FFmpeg runs and two temp files.
Fix
Hash once per file version; convert uploads in one pass; check function signatures once per job.
Result
Less disk and CPU work per job; no stray files.
12
Hygiene

Tests were writing into the real database and storage

Problem
Every test run left job folders in storage and rows in the live database.
Fix
The suite now runs against a throwaway database and folder that are deleted at the end.
Result
80 tests pass and leave nothing behind.

Now listen to the result.