DS-V3 W4FP8 is OOM with 1.0.0rc2, but 1.0.0rc1 is fine

System Info

CPU: x86_64
GPU: Nvidia H20 96GB

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

command:

trtllm-serve serve /data/model \
        --host 0.0.0.0 \
        --port 60000 \
        --backend pytorch \
        --tp_size "8" \
        --pp_size "1" \
        --max_seq_len 32768 \
        --max_num_tokens 32768 \
        --max_batch_size 128 \
        --extra_llm_api_options /data/config/extra-llm-api-config.yml

# extra-llm-api-config.yml
cuda_graph_config:
      padding_enabled: true
      max_batch_size: 128

image: nvcr.io/nvidia/tensorrt-llm/release:1.0.0rc1
image: nvcr.io/nvidia/tensorrt-llm/release:1.0.0rc2

RuntimeError: Executor creation failed due to insufficient GPU memory.

The following component could not be created: Additional executor resources
Total GPU memory (GiB): 95.00
Free GPU memory before component creation attempt (GiB): 16.95

Previously created components and free GPU memory before/after creation (GiB):
Model: 94.30 / 47.83
Sampler: 47.83 / 47.83
Initial KV cache (temporary for KV cache size estimation): 47.83 / 45.67
Drafter: 45.67 / 45.67
Additional executor resources (temporary for KV cache size estimation): 45.67 / 43.59
Model resources created during usage: 43.59 / 45.62
KV cache: 45.68 / 16.95

Please refer to the TensorRT-LLM documentation for information on how to control the memory usage through TensorRT-LLM configuration options. Possible options include:
  Model: reduce max_num_tokens and/or shard the model weights across GPUs by enabling pipeline and/or tensor parallelism
  Sampler: reduce max_seq_len and/or max_attention_window_size
  Initial KV cache (temporary for KV cache size estimation): reduce max_num_tokens
  Drafter: reduce max_seq_len and/or max_draft_len
  Additional executor resources (temporary for KV cache size estimation): reduce max_num_tokens
  Model resources created during usage: reduce max_num_tokens
  KV cache: reduce free_gpu_memory_fraction
  Additional executor resources: reduce max_num_tokens

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/trtllm-serve", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/commands/serve.py", line 302, in serve
    launch_server(host, port, llm_args, metadata_server_cfg, server_role)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/commands/serve.py", line 143, in launch_server
    llm = PyTorchLLM(**llm_args)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 1017, in __init__
    super().__init__(model, tokenizer, tokenizer_mode, skip_tokenizer_init,
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 875, in __init__
    super().__init__(model,
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 203, in __init__
    self._build_model()
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 964, in _build_model
    self._executor = self._executor_cls.create(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/executor.py", line 395, in create
    return GenerationExecutorProxy(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/proxy.py", line 105, in __init__
    self._start_executor_workers(worker_kwargs)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/proxy.py", line 323, in _start_executor_workers
    raise RuntimeError(
RuntimeError: Executor worker returned error

Expected behavior

Not OOM with image: nvcr.io/nvidia/tensorrt-llm/release:1.0.0rc2

actual behavior

OOM while use image: nvcr.io/nvidia/tensorrt-llm/release:1.0.0rc2

additional notes

some behavior change ?