One job at a time
A single-worker queue guarantees two jobs never fight over 6 GB.
This was my first project built around a GPU. Every number on this page was measured on one machine, a gaming laptop, not a cloud server.
The first version ran each chunk in a fresh worker process. That is clean and isolated, but it reloaded the whole model for every chunk. Moving to a single resident model, plus a speaker-conditioning cache keyed by the reference’s content hash, turned model setup from a per-chunk cost into a once-per-session cost.
Mixed precision (fp16/bf16) is the textbook speed-up. I added it behind a setting and benchmarked it on the same chunks and seeds before turning it on:
One job at a time
A single-worker queue guarantees two jobs never fight over 6 GB.
Release once, not always
Clearing the CUDA cache after every chunk slowed things down; once per job is enough.
Unload when idle
After 5 idle minutes the model is unloaded, so the laptop is usable for other work.
Bounded caches
Voice conditionals are kept in a small LRU in memory and capped on disk.
Only text-to-speech generation touches the GPU. Cleaning the reference, checking chunks, joining, loudness and encoding are all fast CPU/NumPy/FFmpeg work. Keeping them off the GPU is a big part of why a 6 GB card is enough.
Next: tuning by measurement.