NVIDIA / NVIDIA/TensorRT-LLM

SimplePool orphans already-created items when preallocation fails partway

Open
#17,909 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Memory
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

System Info
  • TensorRT-LLM: 1.3.0rc23 (built from f20ea652); behaviour re-checked against main @ fbda11f5
  • GPU: NVIDIA GeForce RTX 5090 (sm_120), driver 595.71.05
  • PyTorch 2.11.0+cu130, CUDA 13.0, Python 3.12.13
  • Ubuntu 25.10, kernel 6.17.0-41-generic
  • Backend: PyTorch; single GPU, no TP/PP
  • Model where observed: Qwen3.6-35B-A3B-NVFP4 with FP8 KV cache (where model-specific)
Who can help?

No response

Reproduction

SimplePool builds its preallocated items from a generator. If create_func raises partway through,
_items is never assigned and every item created up to that point is orphaned — for the process-wide
CUDA stream and event pools, up to 128 stream or 1024 event handles.

The failure that triggers this is typically memory pressure, and the leak is re-incurred on every
retry under that same pressure.

Expected behavior

A failed preallocation destroys whatever it already created, leaking nothing.

actual behavior

_items is never assigned and every item created so far is orphaned - up to 128 stream or 1024 event handles - and the leak recurs on each retry under the same memory pressure.

additional notes

Destroy partially-created items when preallocation fails, and make the pools demand-driven so the
preallocation path is not used for the real CUDA pools at all.

We have the patch and 12 CPU-only tests using a plain integer factory (the pool is generic, so the
tests need no GPU): 12 fail against current main, 12 pass with the change.

There is a natural follow-up — RAII for the temporary CUDA stream holder — which we would submit
separately unless you would prefer both as one PR.

Questions for maintainers

  1. Is making the CUDA pools demand-driven acceptable, or is eager preallocation load-bearing
    somewhere we have not found?

Per CONTRIBUTING.md this Issue Request precedes the patch, which is ready and applies cleanly to main @ fbda11f5.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the SimplePool implementation and the 12 CPU-only tests described in the issue; inspect how preallocation handles an exception from the factory. Done means partially created items are cleaned up, the CUDA pools can use demand-driven creation as intended, and all 12 tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.