del model deadlocks on ROCm 7.2.1 + gfx1100 (Windows) — HIP allocator free path
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 4.7k
- Forks
- 537
- Avg merge
- 12h 12m
- Merged PRs (30d)
- 4
Description
Environment
| Item | Value |
|---|---|
| OS | Windows 11 |
| GPU | AMD Radeon RX 7900 XT (gfx1100 / RDNA3, 20 GB VRAM) |
| ROCm | 7.2.1 (Windows wheels) |
| Python | 3.12 |
| PyTorch | 2.9.1+rocm7.2.1 |
| CTranslate2 | 4.7.1 (ROCm build, cp312-cp312-win_amd64) |
| Usage context | faster_whisper / whisperx for ASR transcription |
Symptom
del on the CTranslate2 Whisper model object hangs indefinitely (no CPU/GPU activity, no error, no timeout). The hang occurs during Python reference counting cleanup when the C++ destructor chain runs.
The model loads and transcribes correctly — only teardown hangs.
Minimal Reproduction
import ctranslate2
model = ctranslate2.models.Whisper("large-v3", device="cuda", compute_type="float16")
# ... optional: run some inference ...
del model # HANGS here, never returns
Verified that torch.cuda.empty_cache() (PyTorch's HIP cache cleanup) works fine on the same environment — only CTranslate2's model destructor deadlocks.
Trace
The destructor chain:
Python: del model
→ WhisperModel C++ destructor
→ std::unique_ptr<Layer> members destroyed
→ StorageView::~StorageView() → release() → _allocator->free(data, device)
→ CubCachingAllocator::free() → hipcub::CachingDeviceAllocator::DeviceFree()
OR
CudaAsyncAllocator::free() → hipFreeAsync(ptr, stream)
The hang is in one of these HIP allocator free paths (likely hipcub::DeviceFree or hipFreeAsync), where the ROCm runtime's internal stream synchronization deadlocks on gfx1100.
Workaround
Skip explicit del model in application code — let Python's GC handle cleanup when the function returns. The 4-5 GB model footprint is acceptable on modern GPUs alongside other models (e.g., Wav2Vec2 for alignment).
# DON'T: del model # deadlocks on ROCm
# DO: torch.cuda.empty_cache() # works fine, frees PyTorch cache
# # let model be GC'd at scope exit
Additional Context
torch.cuda.empty_cache()(PyTorch HIP backend) works correctly on the same GPU — no deadlockdelon PyTorch models (e.g., Wav2Vec2) works correctly — only CTranslate2's destructor triggers the issue- The same CTranslate2 version on NVIDIA CUDA does not exhibit this behavior
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the minimal Python reproduction and trace the destructor path through StorageView::release(), CubCachingAllocator::free(), and CudaAsyncAllocator::free(). Compare the hipcub::DeviceFree and hipFreeAsync paths on ROCm 7.2.1 with gfx1100, then verify that model teardown completes without hanging while preserving the reported CUDA behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100