NVIDIA / NVIDIA/TensorRT-LLM

TemporaryCudaStream leaks its borrowed stream when setup or exit fails

Open
#17,914 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Pytorch
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

System Info
  • TensorRT-LLM: 1.3.0rc23 (built from f20ea652); behaviour re-checked against main @ fbda11f5
  • GPU: NVIDIA GeForce RTX 5090 (sm_120), driver 595.71.05
  • PyTorch 2.11.0+cu130, CUDA 13.0, Python 3.12.13
  • Ubuntu 25.10, kernel 6.17.0-41-generic
  • Backend: PyTorch; single GPU, no TP/PP
  • Model where observed: Qwen3.6-35B-A3B-NVFP4 with FP8 KV cache (where model-specific)
Who can help?

No response

Reproduction

TemporaryCudaStream / ItemHolderBase can leak a borrowed CUDA stream back-to-pool on two paths:

  1. Partial construction. ItemHolderBase.__init__ acquires from the pool before self._item is
    established, and TemporaryCudaStream.__init__ can raise in wait_events after super().__init__
    has taken the stream — in both cases the stream is never returned.
  2. Exit. __exit__ records its finish event before returning the stream, so a failure in
    record_event() skips the return entirely.

The leak recurs on every retry under the memory pressure that typically triggers it.

Expected behavior

A borrowed CUDA stream is returned to its pool whether setup and exit succeed or fail.

actual behavior

A raise during construction after acquisition, or a failure in record_event() during exit, skips the return entirely and leaks the stream.

additional notes

Establish self._item = None before acquiring, wrap the post-acquire setup so a failure closes and
returns the stream, and make __exit__ return the stream in a finally so it happens whether or not
the event was recorded.

Five CPU-only tests (the pool is faked, so no GPU is needed): they fail against the exception-safety
baseline and pass with the change.

This stacks on the SimplePool exception-safety fix, which we are raising separately. Happy to submit
them as one two-commit PR if you prefer.

Per CONTRIBUTING.md this Issue Request precedes the patch, which is ready and applies cleanly to main @ fbda11f5.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ItemHolderBase and TemporaryCudaStream, then run the five CPU-only tests described in the issue; the pool is faked, so no GPU is needed. Done means a borrowed stream is returned to its pool when setup raises and when recording the exit event fails, with the existing exception-safety behavior preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.