OpenNMT / OpenNMT/CTranslate2

del model deadlocks on ROCm 7.2.1 + gfx1100 (Windows) — HIP allocator free path

Open
#2,038 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
4.7k
Forks
537
Avg merge
12h 12m
Merged PRs (30d)
4

Description

Environment

Item Value
OS Windows 11
GPU AMD Radeon RX 7900 XT (gfx1100 / RDNA3, 20 GB VRAM)
ROCm 7.2.1 (Windows wheels)
Python 3.12
PyTorch 2.9.1+rocm7.2.1
CTranslate2 4.7.1 (ROCm build, cp312-cp312-win_amd64)
Usage context faster_whisper / whisperx for ASR transcription

Symptom

del on the CTranslate2 Whisper model object hangs indefinitely (no CPU/GPU activity, no error, no timeout). The hang occurs during Python reference counting cleanup when the C++ destructor chain runs.

The model loads and transcribes correctly — only teardown hangs.

Minimal Reproduction

import ctranslate2

model = ctranslate2.models.Whisper("large-v3", device="cuda", compute_type="float16")
# ... optional: run some inference ...
del model  # HANGS here, never returns

Verified that torch.cuda.empty_cache() (PyTorch's HIP cache cleanup) works fine on the same environment — only CTranslate2's model destructor deadlocks.

Trace

The destructor chain:

Python: del model
  → WhisperModel C++ destructor
    → std::unique_ptr<Layer> members destroyed
      → StorageView::~StorageView() → release() → _allocator->free(data, device)
        → CubCachingAllocator::free() → hipcub::CachingDeviceAllocator::DeviceFree()
          OR
          CudaAsyncAllocator::free() → hipFreeAsync(ptr, stream)

The hang is in one of these HIP allocator free paths (likely hipcub::DeviceFree or hipFreeAsync), where the ROCm runtime's internal stream synchronization deadlocks on gfx1100.

Workaround

Skip explicit del model in application code — let Python's GC handle cleanup when the function returns. The 4-5 GB model footprint is acceptable on modern GPUs alongside other models (e.g., Wav2Vec2 for alignment).

# DON'T: del model          # deadlocks on ROCm
# DO:    torch.cuda.empty_cache()  # works fine, frees PyTorch cache
#        # let model be GC'd at scope exit

Additional Context

  • torch.cuda.empty_cache() (PyTorch HIP backend) works correctly on the same GPU — no deadlock
  • del on PyTorch models (e.g., Wav2Vec2) works correctly — only CTranslate2's destructor triggers the issue
  • The same CTranslate2 version on NVIDIA CUDA does not exhibit this behavior

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the minimal Python reproduction and trace the destructor path through StorageView::release(), CubCachingAllocator::free(), and CudaAsyncAllocator::free(). Compare the hipcub::DeviceFree and hipFreeAsync paths on ROCm 7.2.1 with gfx1100, then verify that model teardown completes without hanging while preserving the reported CUDA behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.