[vLLM] Missing --max-model-len guard for offload-disabled agentic runs causes KV cache OOM on H100 / [vLLM] 禁用 offload 的 agentic 运行缺少 --max-model-len 保护导致 H100 上 KV 缓存 OOM

Open Beginner friendly
#1,587 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
68/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python
Domain
machine-learning

Research direction

Start with benchmarks/single_node/agentic/kimik2.5_int4_h100.sh at line 4 and .github/configs/nvidia-master.yaml at line 9195, then reproduce the offload-disabled launch without --max-model-len. Update the launch handling so the server avoids the reported KV-cache startup failure, and verify that the benchmark starts successfully on the stated 8×H100 configuration.

Written by the indexing model from the issue text.

Description

Describe the bug
When running Kimi-K2.5 INT4 agentic benchmarks with vLLM on 8×H100 (TP=8) without CPU offloading, the engine crashes immediately at startup due to insufficient KV cache memory. The server never starts and all TP workers are terminated.

https://github.com/SemiAnalysisAI/InferenceX/blob/c9798a7708826dd4ead79acad4500000f72d5576/benchmarks/single_node/agentic/kimik2.5_int4_h100.sh#L4

It is triggered in the master file here.
https://github.com/SemiAnalysisAI/InferenceX/blob/c9798a7708826dd4ead79acad4500000f72d5576/.github/configs/nvidia-master.yaml#L9195

To Reproduce

  1. Launch a vLLM agentic benchmark for Kimi-K2.5 INT4 on 8×H100 with TP=8
  2. Set offloading to none (no CPU KV cache offloading)
  3. Do not set --max-model-len in the vllm serve command
  4. Observe engine crash at startup

Expected behavior
The vLLM server should start successfully and serve agentic requests. Either the launch script should cap --max-model-len to a value that fits in available VRAM when offloading is disabled, or the error
should be surfaced earlier with a clear recommendation.

Screenshots
N/A

Additional context
Root cause: without --max-model-len, vLLM defaults to the model's full context length (262,144 tokens), which requires 8.58 GiB of KV cache. On 8×H100 with TP=8, after loading the INT4 weights, only ~0.9 GiB
of VRAM remains for KV cache.

Error

ValueError: To serve at least one request with the model's max seq len (262144),
(8.58 GiB KV cache is needed, which is larger than the available KV cache memory
(0.9 GiB). Based on the available memory, the estimated maximum model length is 27392.
Try increasing gpu_memory_utilization or decreasing max_model_len when initializing
the engine.

中文说明

在 8×H100（TP=8）上使用 vLLM 运行 Kimi-K2.5 INT4 agentic 基准测试时，未启用 CPU offloading 且未设置 --max-model-len，导致 vLLM 默认使用模型完整上下文长度（262,144 tokens），所需 KV 缓存（8.58 GiB）超过加载 INT4 权重后的可用显存（约 0.9 GiB），引擎启动时立即崩溃。应在启动脚本中为禁用 offloading 的运行设置 --max-model-len 上限。

Dominant language
Python
Stars
1.7k
Forks
303
Avg merge
1d 13h
Merged PRs (30d)
284

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from SemiAnalysisAI/InferenceX

All issues in SemiAnalysisAI/InferenceX

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.