TemporaryCudaStream leaks its borrowed stream when setup or exit fails
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
System Info
- TensorRT-LLM: 1.3.0rc23 (built from
f20ea652); behaviour re-checked againstmain@fbda11f5 - GPU: NVIDIA GeForce RTX 5090 (sm_120), driver 595.71.05
- PyTorch 2.11.0+cu130, CUDA 13.0, Python 3.12.13
- Ubuntu 25.10, kernel 6.17.0-41-generic
- Backend: PyTorch; single GPU, no TP/PP
- Model where observed: Qwen3.6-35B-A3B-NVFP4 with FP8 KV cache (where model-specific)
Who can help?
No response
Reproduction
TemporaryCudaStream / ItemHolderBase can leak a borrowed CUDA stream back-to-pool on two paths:
- Partial construction.
ItemHolderBase.__init__acquires from the pool beforeself._itemis
established, andTemporaryCudaStream.__init__can raise inwait_eventsaftersuper().__init__
has taken the stream — in both cases the stream is never returned. - Exit.
__exit__records its finish event before returning the stream, so a failure in
record_event()skips the return entirely.
The leak recurs on every retry under the memory pressure that typically triggers it.
Expected behavior
A borrowed CUDA stream is returned to its pool whether setup and exit succeed or fail.
actual behavior
A raise during construction after acquisition, or a failure in record_event() during exit, skips the return entirely and leaks the stream.
additional notes
Establish self._item = None before acquiring, wrap the post-acquire setup so a failure closes and
returns the stream, and make __exit__ return the stream in a finally so it happens whether or not
the event was recorded.
Five CPU-only tests (the pool is faked, so no GPU is needed): they fail against the exception-safety
baseline and pass with the change.
This stacks on the SimplePool exception-safety fix, which we are raising separately. Happy to submit
them as one two-commit PR if you prefer.
Per CONTRIBUTING.md this Issue Request precedes the patch, which is ready and applies cleanly to main @ fbda11f5.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ItemHolderBase and TemporaryCudaStream, then run the five CPU-only tests described in the issue; the pool is faked, so no GPU is needed. Done means a borrowed stream is returned to its pool when setup raises and when recording the exit event fails, with the existing exception-safety behavior preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- backend, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100