SimplePool orphans already-created items when preallocation fails partway
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
System Info
- TensorRT-LLM: 1.3.0rc23 (built from
f20ea652); behaviour re-checked againstmain@fbda11f5 - GPU: NVIDIA GeForce RTX 5090 (sm_120), driver 595.71.05
- PyTorch 2.11.0+cu130, CUDA 13.0, Python 3.12.13
- Ubuntu 25.10, kernel 6.17.0-41-generic
- Backend: PyTorch; single GPU, no TP/PP
- Model where observed: Qwen3.6-35B-A3B-NVFP4 with FP8 KV cache (where model-specific)
Who can help?
No response
Reproduction
SimplePool builds its preallocated items from a generator. If create_func raises partway through,
_items is never assigned and every item created up to that point is orphaned — for the process-wide
CUDA stream and event pools, up to 128 stream or 1024 event handles.
The failure that triggers this is typically memory pressure, and the leak is re-incurred on every
retry under that same pressure.
Expected behavior
A failed preallocation destroys whatever it already created, leaking nothing.
actual behavior
_items is never assigned and every item created so far is orphaned - up to 128 stream or 1024 event handles - and the leak recurs on each retry under the same memory pressure.
additional notes
Destroy partially-created items when preallocation fails, and make the pools demand-driven so the
preallocation path is not used for the real CUDA pools at all.
We have the patch and 12 CPU-only tests using a plain integer factory (the pool is generic, so the
tests need no GPU): 12 fail against current main, 12 pass with the change.
There is a natural follow-up — RAII for the temporary CUDA stream holder — which we would submit
separately unless you would prefer both as one PR.
Questions for maintainers
- Is making the CUDA pools demand-driven acceptable, or is eager preallocation load-bearing
somewhere we have not found?
Per CONTRIBUTING.md this Issue Request precedes the patch, which is ready and applies cleanly to main @ fbda11f5.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the SimplePool implementation and the 12 CPU-only tests described in the issue; inspect how preallocation handles an exception from the factory. Done means partially created items are cleaned up, the CUDA pools can use demand-driven creation as intended, and all 12 tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100