Discovered vibevoice sitting at 3.2GB resident GPU memory with
loaded_models: [] and no idle-unload ever firing for it again. Root cause:
a load attempt had OOM'd partway through (competing with an unrelated
ollama process on the same GPU), so _MODEL_CACHE was never populated —
the idle-unload loop only clears that cache, so it had nothing to act on,
even though the partially-constructed model had already left memory
allocated. gc.collect()+empty_cache() don't reliably reclaim memory from
an interrupted from_pretrained() call.
Confirmed a plain process restart does fully reclaim it, so each worker
now treats any load failure as fatal: log it and os._exit(1), letting
supervisord's autorestart=true respawn a clean process immediately.
Verified with a bogus model id — worker exits, respawns, and passes the
smoke test right after.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every ASR/TTS worker (faster-whisper, qwen3, vibevoice, xtts) kept every
model it ever loaded resident in GPU memory forever, and switching to a
different model (e.g. a different whisper size) just added another one
alongside it rather than freeing the old one. Combined with the shared
24GB GPU, this made memory pressure only ever go up.
Now: only one model stays resident per worker at a time (loading a
different model_id evicts the previous one first), and the whole cache
(plus, for faster-whisper, the diarization pipeline) is dropped after 2
minutes of no requests. Verified end-to-end: the idle timer actually fires
and reclaims memory, and switching qwen3 models evicts the old one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Third ASR backend option alongside faster-whisper and qwen3. Builds
microsoft/VibeVoice from source (pinned to a specific commit, since it's
custom modeling code not in transformers' Auto* registry) into its own venv,
inheriting the base image's torch/CUDA. Comes with built-in speaker
diarization (VibeVoiceASRProcessor.post_process_transcription returns
per-segment speaker ids directly, no separate pyannote pass needed).
Verified end-to-end against a real recording: 200 OK, correct Korean
transcription, speaker labels populated.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
