[Bug]: trtllm-serve periodically hanging with "No free slots"

System Info

Ubuntu 24.04.2 LTS
TensorRT-LLM 1.2.0rc3 (tried also 1.2.0rc1, 1.2.0rc2)
x86
CUDA 12.8
B200-SXM
Nvidia Driver 570.172.08

Who can help?

@DanBlanaru

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Steps to reproduce:
Start tensorrt-llm 1.2.0rc3 container with args:

        - trtllm-serve
        - serve
        - nvidia/DeepSeek-V3-0324-FP4
        - '--host'
        - 0.0.0.0
        - '--port'
        - '8000'
        - '--tp_size'
        - '8'
        - '--gpus_per_node'
        - '8'
        - '--backend'
        - pytorch
        - '--extra_llm_api_options'
        - /config/extra-llm-api-config.yml
        - '--max_batch_size'
        - '16'
        - '--max_seq_len'
        - '32768'
        - '--max_num_tokens'
        - '32768'
        - '--ep_size'
        - '2'
        - '--kv_cache_free_gpu_memory_fraction'
        - '0.8'

/config/extra-llm-api-config.yml:

    cuda_graph_config:
      batch_sizes:
      - 1
      - 2
      - 3
      - 4
      - 5
      - 6
      - 7
      - 8
      - 9
      - 10
      - 11
      - 12
      - 13
      - 14
      - 15
      - 16
      enable_padding: true
    enable_iter_perf_stats: true
    enable_iter_req_stats: true
    guided_decoding_backend: xgrammar
    kv_cache_config:
      dtype: fp8
      free_gpu_memory_fraction: 0.8
    moe_config:
      backend: TRTLLM
    print_iter_log: true
    return_perf_metrics: true
    speculative_config:
      decoding_type: MTP
      num_nextn_predict_layers: 3
    stream_interval: 8

Leave under load (cannot share exact dataset, because requests are coming from different clients with various parameters).

Expected behavior

Container works normally

actual behavior

After some period it crashes with

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 343, in _event_loop_wrapper
    self.event_loop()
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1423, in _executor_loop_overlap
    self.event_loop()
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/resource_manager.py", line 1339, in prepare_resources
    self.resource_manager.prepare_resources(scheduled_batch)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/seq_slot_manager.py", line 25, in prepare_resources
    resource_manager.prepare_resources(scheduled_batch)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/resource_manager.py", line 1227, in add_slot
    llm_req.seq_slot = self.slot_manager.add_slot(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    raise ValueError("No free slots")

ValueError: No free slots

additional notes

The problem reproduces very stable (e.g. failure after 1-2 hours running). Is not reproducible when sending small requests (<8000 tokens). Failures seems random, not related to a single specific request.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.