How it works
The big picture
Section titled “The big picture”A request flows through four layers: API → controller → job queue → pipeline services. The GPU is a shared, scarce resource, so exactly one synthesis job runs at a time, and the model stays loaded between chunks and jobs.
- 1Script cleanupCPU
Punctuation, abbreviations, markdown, numbers
- 2Smart chunkingCPUimproved
Sentence & Danda (।) aware, ≤220 chars; mid-sentence cuts keep a comma
- 3Voice conditioning clipCPUimproved
De-clip, de-rumble, light denoise, pick the best 10 s window
- 4Chatterbox synthesisGPU
Resident model, cached voice conditioning, tuned settings
- 5Chunk checkCPUimproved
Babble, truncation and dead-air detection; one retry with a new seed
- 6Natural joiningCPUimproved
Trim, level-match, pauses by punctuation, filled with room tone
- 7Polish & labelCPUimproved
Two-pass loudness to −16 LUFS + AI-attribution tags in one encode
- 8DeliverCPU
Exactly two files: WAV master + MP3
Stage by stage
Section titled “Stage by stage”-
Script cleanup. Smart quotes, dashes, markdown leftovers and repeated punctuation are normalised; English abbreviations, months and currencies are expanded so the model never has to guess how to say
$5orDr.. -
Chunking. Chatterbox works best on short passages, so text is split on sentence ends, including the Hindi Danda
।and double Danda॥, while skipping abbreviations and decimals. Chunks stay under ~220 characters. If a long sentence must be cut, the cut lands on a comma or space, and the chunk gets a trailing comma so the voice keeps a continuing intonation instead of a full stop. -
Voice conditioning. The reference recording is cleaned (clipping repaired, rumble removed, light denoise) and the best 10-second window is moved to the front, because the model only looks at the first 6–10 seconds. The result is cached by content hash, so this happens once per voice.
-
Synthesis. The resident Chatterbox model generates each chunk with tuned expressiveness, guidance and temperature. Progress, ETA and resumable chunk files are tracked per job.
-
Chunk check. A cheap, GPU-free check catches the usual failure modes: babbling (far too slow for the text), truncation (far too fast), the 40-second generation cap, or long dead air. A failed chunk is regenerated once with a new seed, and the better take is kept.
-
Natural joining. Each chunk’s own silence is trimmed, loudness is matched, and pauses are sized by punctuation: commas short, questions a bit longer, paragraphs longest, with slight variation. Pauses are filled with the recording’s own room tone, never digital silence.
-
Polish & label. Loudness is measured once, then a single fixed gain brings the whole file to −16 LUFS, without pumping. The same encode writes AI-attribution metadata.
-
Deliver. The job folder ends with exactly two files: a WAV master and an MP3. Chunks, temp files and the uploaded reference are removed.
Under the hood
Section titled “Under the hood”| Layer | Technology |
|---|---|
| API | FastAPI, Pydantic v2, security headers, per-IP rate limits |
| Persistence | SQLAlchemy 2.0 + SQLite (jobs, chunk progress, voices) |
| Queue | Single-worker GPU-safe FIFO with timeouts and an idle janitor |
| Model | Chatterbox Multilingual (Resemble AI), PyTorch 2.6 + CUDA 12.4 |
| Audio | FFmpeg, NumPy, SoundFile, librosa |
| Frontend | React 19 + TypeScript + Vite |
| Tests | pytest, 80 tests, isolated temp DB and storage |
POST /v1/ttsvalidates text, language and voice, checks quotas, creates a job.- The job enters the single GPU queue; the UI polls
GET /v1/jobs/{id}for chunk progress and ETA. - If interrupted,
POST /v1/jobs/{id}/resumecontinues from the last finished chunk. GET /v1/audio/{id}/output?format=mp3streams the result.
- One GPU job at a time; the model stays resident, with an idle unload after 5 minutes.
- No junk: every stage cleans up after itself; tests run in a throwaway folder.
- Every output is labelled as AI-generated and carries an inaudible watermark.
- Filesystem paths are never exposed by the public API.