[Bug]: Multimodal Model Fails in Multi-turn Dialogue with Mixed Message Types

System Info

• TensorRT-LLM version: 1.1.0rc5 and main (maybe all release version)

Who can help?

@laikhtewari

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

• Serving a multimodal model with TensorRT-LLM

trtllm-serve \
  /path/to/mm_model \
  --host 0.0.0.0 --port 8000 \
  --max_batch_size 4 --max_seq_len 16384 \
  --max_num_tokens 4096 --tp_size 2 --pp_size 1 \
  --kv_cache_free_gpu_memory_fraction 0.95

• Prepare a conversation with mixed message types and send to the server

messages = [
    {
        "role": "user",
        "content": "What's in this image?",
        "media": [{"modality": "image", "data": "image_data_here"}]
    },
    {
        "role": "assistant", 
        "content": "I see a cat in the image."
    },
    {
        "role": "user",
        "content": "Can you describe it in more detail?"  # Text-only message
    }
]

Expected behavior

The model should correctly identify which turns actually contain multimodal content and apply special tokens/processing only to those turns, leaving text-only turns unaffected.

actual behavior

• The preprocessing logic, which uses a global multimodal input count, erroneously inserts multimodal special tokens into turns that contain only text.
• The visual processor receives a text string that now contains these special tokens. It attempts to locate and process image patches or other multimodal data based on these tokens but fails because no actual image data was provided for that specific turn.

additional notes

The problem stems from the global multimodal data tracker that accumulates data across all messages rather than isolating processing to individual messages.
tensorrt_llm/serve/chat_utils.py#L175

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Start at tensorrt_llm/serve/chat_utils.py#L175 and run the trtllm-serve reproduction with the mixed conversation shown in the issue. Trace how multimodal data is tracked across turns. Done means multimodal processing and special tokens apply only to turns containing multimodal content, while the text-only turn remains unaffected.