Skip to content

Built on my first GPU

This was my first project built around a GPU. Every number on this page was measured on one machine, a gaming laptop, not a cloud server.

RTX 3050
Laptop GPU
6 GB VRAM · ~35 W power limit
16 GB
System RAM
Intel Core i5-12450HX, 8 cores
3.8 GB
Peak VRAM in use
model + activations, per chunk
≈1.3×
Real-time factor
seconds of compute per second of audio, warm

The first version ran each chunk in a fresh worker process. That is clean and isolated, but it reloaded the whole model for every chunk. Moving to a single resident model, plus a speaker-conditioning cache keyed by the reference’s content hash, turned model setup from a per-chunk cost into a once-per-session cost.

Seconds of compute per second of audiolower is better
  • Per-chunk worker (v0)≈4.4×
  • Resident model, warm≈1.3×

v0 figure from the project’s early benchmark log; resident figure from a 7-chunk end-to-end job (first chunk excluded, since it includes warm-up).

Lesson 2: “faster” settings can be slower

Section titled “Lesson 2: “faster” settings can be slower”

Mixed precision (fp16/bf16) is the textbook speed-up. I added it behind a setting and benchmarked it on the same chunks and seeds before turning it on:

Real-time factor by precision (4 Hindi chunks, 47 s of audio)lower is better
  • Full precision1.38×
  • bf16 autocast2.83×
  • fp16 autocast3.34×

Mixed precision also raised peak VRAM from 3.85 GB to 4.82 GB. The option was removed entirely, not just turned off.

One job at a time

A single-worker queue guarantees two jobs never fight over 6 GB.

Release once, not always

Clearing the CUDA cache after every chunk slowed things down; once per job is enough.

Unload when idle

After 5 idle minutes the model is unloaded, so the laptop is usable for other work.

Bounded caches

Voice conditionals are kept in a small LRU in memory and capped on disk.

Only text-to-speech generation touches the GPU. Cleaning the reference, checking chunks, joining, loudness and encoding are all fast CPU/NumPy/FFmpeg work. Keeping them off the GPU is a big part of why a 6 GB card is enough.

Next: tuning by measurement.