[Bug] trtllm-serve Phi-4-MM: default kv_cache_config.enable_block_reuse=True breaks 2nd multimodal request ("Multimodal token count mismatch")

System Info

• TensorRT-LLM version: 1.2.0rc6
• Container image: nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc6
• Model: nvidia/Phi-4-multimodal-instruct-NVFP4
• Endpoint: POST /v1/chat/completions
• GPU: NVIDIA RTX 5080 (Blackwell, sm_120), single GPU
• Server config: --max_seq_len 8192 --kv_cache_free_gpu_memory_fraction 0.5 --trust_remote_code (everything else default — i.e. KvCacheConfig(enable_block_reuse=True))
• Audio sample: 16 s mono 16 kHz PCM WAV at /data/sample.wav (via bind-mount)

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Start the server (default flags, KV-block reuse on):

docker run --gpus all --rm -p 8000:8000 --ipc=host \
  -e HF_HOME=/root/.cache/huggingface \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  -v $PWD/host_data:/data:ro \
  nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc6 \
  trtllm-serve nvidia/Phi-4-multimodal-instruct-NVFP4 \
    --host 0.0.0.0 --port 8000 \
    --trust_remote_code \
    --max_seq_len 8192 \
    --kv_cache_free_gpu_memory_fraction 0.5

Confirm in the startup log:

... kv_cache_config=KvCacheConfig(enable_block_reuse=True, ...) ...

Save this request payload to req.json:

{
  "model": "nvidia/Phi-4-multimodal-instruct-NVFP4",
  "messages": [{"role": "user", "content": [
    {"type": "text", "text": "Transcribe the audio into Russian text."},
    {"type": "audio_url", "audio_url": {"url": "/data/sample.wav"}}
  ]}],
  "max_tokens": 64, "temperature": 0
}

Send it 3 times in a row, same payload, no other traffic in between:

for i in 1 2 3; do
  echo "=== R$i ==="
  curl -sS --max-time 60 -X POST http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" --data-binary "@req.json" \
    -w "\nHTTP %{http_code} time=%{time_total}s\n"
done

Result on this exact setup (just verified on 1.2.0rc6):

=== R1 ===
{"id":"chatcmpl-...","choices":[{"message":{"role":"assistant","content":"..."}}]}
HTTP 200 time=1.16s

=== R2 ===
{"object":"error","message":"Multimodal token count mismatch: found 0 image tokens in input_ids but received 198 image embeddings. This is likely due to KV cache reuse, chunk prefill, or other optimizations that cause token count mismatches within the inference batch.","type":"BadRequestError","code":400}
HTTP 400 time=0.13s

=== R3 ===
curl: (28) Operation timed out after 60010 milliseconds with 0 bytes received
HTTP 000 time=60.01s

So the very first request succeeds; the second identical request returns the "Multimodal token count mismatch ... This is likely due to KV cache reuse" error; subsequent requests hang. The error message itself diagnoses the cause.

Verified workaround

Pass --extra_llm_api_options with kv_cache_config.enable_block_reuse: false. Create llm_args.yaml:

kv_cache_config:
  enable_block_reuse: false

Restart the server with --extra_llm_api_options /workspace/llm_args.yaml. Startup log now shows KvCacheConfig(enable_block_reuse=False, ...). Running the same 5 identical requests in a row now all return HTTP 200, consistent output, no hangs.

Expected behavior

Either:

1. kv_cache_config.enable_block_reuse should default to False whenever a multimodal model is loaded, or
2. The cache key for prefix-block reuse should include the multimodal-embedding hash (so the second request's cached prefix doesn't collide with the freshly-computed audio embeddings), or
3. At minimum, surface the existing --extra_llm_api_options workaround in the trtllm-serve --help output and in the docs page for multimodal serving — right now users have no way to discover it without reading source.

The first request shouldn't be an only-shot situation. Batch ASR / vision pipelines need stable behavior across thousands of requests.

actual behavior

Default enable_block_reuse=True plus a multimodal model = exactly one good request, then the cache mismatch fires and the server is stuck until restart.

additional notes

Source — verified on nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc6

tensorrt_llm/llmapi/llm_args.py:1562 — the default:

class KvCacheConfig(StrictBaseModel, PybindMirror):
    enable_block_reuse: bool = Field(
        default=True,
        description="Controls if KV cache blocks can be reused for different requests.")

tensorrt_llm/commands/serve.py:117-119 — trtllm-serve constructs KvCacheConfig with only free_gpu_memory_fraction, no CLI option for enable_block_reuse:

kv_cache_config = KvCacheConfig(
    free_gpu_memory_fraction=free_gpu_memory_fraction,
)

tensorrt_llm/commands/serve.py:362-370 — --extra_llm_api_options (alias --config) is the only way to override:

@click.option(
    "--config",
    "--extra_llm_api_options",
    "extra_llm_api_options",
    type=str,
    default=None,
    help="Path to a YAML file that overwrites the parameters specified by trtllm-serve. "
         "Can be specified as either --config or --extra_llm_api_options.")

What the error message says (and why I think it's accurate)

The error literally contains "This is likely due to KV cache reuse, chunk prefill, or other optimizations that cause token count mismatches within the inference batch." Disabling enable_block_reuse (the only one of those three knobs that's on by default in trtllm-serve) makes the error disappear and lets the server handle arbitrarily many identical multimodal requests in a row. That points cleanly at block reuse as the culprit, and at the cache key not capturing the multimodal embeddings.

Suggested fix

The minimal, low-risk fix: in tensorrt_llm/commands/serve.py:117 (or wherever KvCacheConfig lands), detect whether the model being loaded has any registered multimodal input processor (the registry already exists for that — see tensorrt_llm/inputs/registry.py), and if yes, set enable_block_reuse=False by default. Users who explicitly pass --extra_llm_api_options kv_cache_config.enable_block_reuse=true can opt back in once the proper fix lands.

The proper fix: include the multimodal-embedding hash in the prefix-block cache key so reuse becomes safe for multimodal models too. That's a larger change inside the C++ KV cache manager.

Related

• Companion issue: #14100 — audio_url doesn't decode data: or file:// URIs, leaving audios=None reaching the processor. That bug hides this one under a different symptom: when audio fails to load, you also get a "token count mismatch" error message because N image embeddings are already allocated against zero placeholders. After PR #14010 lands and unmasks audio loading, this KV-cache reuse bug becomes the next failure mode for any serving setup that sends more than one identical multimodal request.

I'm happy to put together a PR for the multimodal-detect default flip if that direction is welcome.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Reproduce the repeated-request failure with trtllm-serve and the documented curl loop, then inspect tensorrt_llm/commands/serve.py, tensorrt_llm/llmapi/llm_args.py, and the multimodal registry at tensorrt_llm/inputs/registry.py. Done means repeated identical multimodal requests remain successful without hangs, while the supported configuration and any default change are documented or covered by an appropriate regression check.