NVIDIA / NVIDIA/TensorRT-LLM

[Bug]: Memory leak: Qwen3 30B A3B pytorch backend

Open
#13,777 1 comment 0 reactions 1 assignee View on GitHub

@hchings is already working on this.

Since May 12, 2026.

bug Memory Pytorch
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

System Info
  • CPU architecture: amd64
  • CPU memory size: 80Gb
  • GPU properties
    • GPU name: NVIDIA H200
    • GPU memory size: 71424MiB (4g.71gb MIG)
    • Driver Version: 580.126.20
    • CUDA Version: 13.1
  • TensorRT-LLM docker image used: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc12
Who can help?

No response

Information
  • The official example scripts
  • My own modified scripts
Tasks
  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)
Reproduction

Steps to reproduce behaviour:

cat <<'EOF' > config.yaml
max_seq_len: 32768
max_num_tokens: 16384
max_batch_size: 64

num_postprocess_workers: 4

stream_interval: 5

enable_chunked_prefill: true

return_perf_metrics: true
enable_iter_perf_stats: true
enable_iter_req_stats: true

cuda_graph_config:
  enable_padding: true
  max_batch_size: 32
EOF
  1. trtllm-serve Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --config=config.yaml
  2. for i in $(seq 1 100000); do curl -s -X POST http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"Say '\''ok'\''"}],"model":"dsp.optimized.qwen3-30b-a3b","max_tokens":512,"temperature":1e-05}'; echo; done
Expected behavior

Container memory (RAM) usage shouldn't keep increasing as requests are served.

actual behavior

Container memory usage keeps growing as requests are served till it's OOMKilled.

additional notes
  • The memory usage remains flat when there is no traffic.
  • This happens with 1.3.0rc13 as well.
Before submitting a new issue...
  • Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.