[Bug]: Gemma 3 / Gemma 4 fail in TensorRT-LLM 1.3.0rc17 container on NVIDIA L4: FlashInfer unsupported architecture, kv_layout metadata error, AutoDeploy use_cache mismatch

System Info

System Info

• CPU architecture:

  x86_64
  

• CPU / Host memory:

                 total        used        free      shared  buff/cache   available
  Mem:            15Gi       872Mi       4.0Gi       0.0Ki        10Gi        14Gi
  Swap:             0B          0B          0B
  

• GPU:

  NVIDIA L4
  

• GPU compute capability:

  8.9
  

• GPU memory:

  TensorRT-LLM logs report:
  GPU 0 has a total capacity of 21.95 GiB
  

• GPU clocks:

  Driver Version: 550.90.07
  CUDA Version: 12.4
  
  Graphics Clock: 1020 MHz
  SM Clock:       1020 MHz
  Memory Clock:   6250 MHz
  Video Clock:    945 MHz
  
  Max Graphics Clock: 2040 MHz
  Max SM Clock:       2040 MHz
  Max Memory Clock:   6251 MHz
  Max Video Clock:    1770 MHz
  

• Container image:

  nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc17
  

• TensorRT-LLM version from logs:

  [TensorRT-LLM] TensorRT LLM version: 1.3.0rc17
  

• Python:

  Python 3.12 inside container
  

• PyTorch version from logs:

  torch version 2.11.0a0+eb65b36914.nv26.02
  

• NVIDIA driver:

  550.90.07
  

• CUDA reported by nvidia-smi:

  CUDA Version: 12.4
  

• Transformers version from warning:

  transformers version 5.5.4
  

• ModelOpt warning from container:

  /usr/local/lib/python3.12/dist-packages/modelopt/torch/__init__.py:36:
  UserWarning: transformers version 5.5.4 is incompatible with nvidia-modelopt and may cause issues.
  Please install recommended version with `pip install nvidia-modelopt[hf]` if working with HF models.
  

• torchao warning from container:

  Skipping import of cpp extensions due to incompatible torch version 2.11.0a0+eb65b36914.nv26.02 for torchao version 0.15.0
  

• Runtime:

  Docker container with --gpus all
  

• GPU setup:

  Single NVIDIA L4 GPU
  

• Backends tested:

  pytorch
  _autodeploy
  

• Models tested:

  google/gemma-3-4b-it
  google/gemma-4-E2B-it
  google/gemma-4-E4B-it
  

• Commands used to collect system information:

  uname -m
  free -h
  nvidia-smi --query-gpu=name,compute_cap --format=csv
  nvidia-smi -q -d CLOCK
  

``

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Reproduction

I am using the official NVIDIA TensorRT-LLM container:

nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc17

on a single NVIDIA L4 GPU. I attempted to serve documented Gemma models using trtllm-serve. I encountered multiple failures depending on model/backend.

Before running the commands, I enabled debug logging:

export TLLM_DEBUG_MODE=1

• Gemma 3 with the PyTorch backend fails with missing kv_layout.

  Command:

  trtllm-serve serve google/gemma-3-4b-it \
    --host 0.0.0.0 \
    --port 8000 \
    --max_batch_size 16 \
    --max_num_tokens 500 \
    --max_seq_len 4096
  

  Observed failure:

  AttributeError: 'TrtllmAttentionMetadata' object has no attribute 'kv_layout'
  

  Relevant traceback:

  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/attention_backend/flashinfer.py", line 1518, in forward_impl
      self.layer_idx, kv_layout=metadata.kv_layout)
                                ^^^^^^^^^^^^^^^^^^
  AttributeError: 'TrtllmAttentionMetadata' object has no attribute 'kv_layout'
  

  This happens after the model is downloaded, initialized, weights are loaded, KV cache is allocated, and warmup begins.

• Gemma 4 E2B fails with FlashInfer unsupported architecture on NVIDIA L4.

  Command:

  trtllm-serve serve google/gemma-4-E2B-it \
    --host 0.0.0.0 \
    --port 8000 \
    --max_batch_size 8 \
    --max_num_tokens 250 \
    --max_seq_len 4096
  

  Important log line:

  [TRT-LLM] [I] [_torch] Applied model defaults for Gemma4ForConditionalGeneration: {'attn_backend': 'FLASHINFER'}
  

  Then FlashInfer dynamically downloads cubins:

  flashinfer.jit: Fetching ... fmha/trtllm-gen ...
  

  Observed failure:

  RuntimeError: Error in function 'TllmGenFmhaRunner'
  at /usr/local/lib/python3.12/dist-packages/flashinfer/data/include/flashinfer/trtllm/fmha/fmhaRunner.cuh:37:
  Unsupported architecture
  

  Relevant traceback:

  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/attention_backend/flashinfer.py", line 1562, in prefill_forward
      wrapper.run(q[:num_ctx_tokens],
  ...
  File "/usr/local/lib/python3.12/dist-packages/flashinfer/prefill.py", line 264, in _paged_run
      op.trtllm_paged_attention_context(
  ...
  RuntimeError: Error in function 'TllmGenFmhaRunner' ... Unsupported architecture
  

  This appears to happen because Gemma 4 automatically selects FLASHINFER, but the selected FlashInfer TRTLLM FMHA runner does not support NVIDIA L4 / SM89.

• Gemma 3 with the AutoDeploy backend fails with a use_cache constructor mismatch.

  Command:

  trtllm-serve serve google/gemma-3-4b-it --backend _autodeploy
  

  Observed failure:

  TypeError: Gemma3ForConditionalGeneration.__init__() got an unexpected keyword argument 'use_cache'
  

  Relevant traceback:

  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/auto_deploy/models/hf.py", line 281, in _build_model
      model = self.automodel_cls.from_config(
  ...
  File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 1498, in _from_config
      model = cls(config, **kwargs)
  TypeError: Gemma3ForConditionalGeneration.__init__() got an unexpected keyword argument 'use_cache'
  

  This appears to be a compatibility issue between TensorRT-LLM AutoDeploy and the bundled Hugging Face Transformers Gemma 3 implementation.

• Gemma 4 E4B fails with CUDA OOM on NVIDIA L4.

  Command:

  trtllm-serve serve google/gemma-4-E4B-it \
    --host 0.0.0.0 \
    --port 8000 \
    --max_batch_size 16 \
    --max_num_tokens 500 \
    --max_seq_len 4096
  

  Observed memory logs:

  [TRT-LLM] [I] [_torch] Use 14.79 GB for model weights.
  

  Observed failure:

  CUDA out of memory. Tried to allocate 5.25 GiB.
  GPU 0 has a total capacity of 21.95 GiB of which 3.56 GiB is free.
  Of the allocated memory 16.78 GiB is allocated by PyTorch,
  and 1.16 GiB is reserved by PyTorch but unallocated.
  

  Final TensorRT-LLM error:

  RuntimeError: Executor creation failed due to insufficient GPU memory.
  
  The following component could not be created: Model
  Total GPU memory (GiB): 21.95
  Free GPU memory before component creation attempt (GiB): 21.56
  

  This may be expected due to Gemma 4 multimodal components and L4 memory limits, but the supported-model documentation does not clearly state minimum GPU memory requirements or whether Gemma 4 E4B can run on L4.

Expected behavior

Expected behavior

Since Gemma 3 and Gemma 4 are listed as supported model families/classes in the TensorRT-LLM documentation, I expected one of the following behaviors:

1. google/gemma-3-4b-it should serve successfully with trtllm-serve on NVIDIA L4 using the PyTorch backend.

2. If a specific Gemma model/backend/GPU combination is unsupported, TensorRT-LLM should fail early with a clear actionable error, for example:

   Gemma4ForConditionalGeneration with FLASHINFER is not supported on NVIDIA L4 / SM89.
   Please use backend X or configuration Y.
   

3. For Gemma 4 on L4, TensorRT-LLM should not automatically select FLASHINFER if the selected FlashInfer FMHA runner does not support NVIDIA L4 / SM89.

4. AutoDeploy should either support Gemma 3 with the bundled Transformers version or fail early with a documented Transformers version compatibility error.

5. If Gemma 4 E2B/E4B require more memory than NVIDIA L4 provides, the documentation should provide clear minimum GPU memory requirements and recommended settings.

6. If Gemma 4 multimodal models cannot be served in text-only mode without loading vision components, this limitation should be documented.

actual behavior

Actual behavior

Multiple Gemma models fail in different ways on the official TensorRT-LLM 1.3.0rc17 container with NVIDIA L4:

1. google/gemma-3-4b-it with the PyTorch backend fails during warmup:

   AttributeError: 'TrtllmAttentionMetadata' object has no attribute 'kv_layout'
   

2. google/gemma-4-E2B-it automatically selects FLASHINFER:

   Applied model defaults for Gemma4ForConditionalGeneration: {'attn_backend': 'FLASHINFER'}
   

   Then it fails during warmup:

   RuntimeError: Error in function 'TllmGenFmhaRunner' ... Unsupported architecture
   

3. google/gemma-3-4b-it --backend _autodeploy fails during model construction:

   TypeError: Gemma3ForConditionalGeneration.__init__() got an unexpected keyword argument 'use_cache'
   

4. google/gemma-4-E4B-it fails with CUDA OOM on NVIDIA L4:

   CUDA out of memory. Tried to allocate 5.25 GiB.
   GPU 0 has a total capacity of 21.95 GiB of which 3.56 GiB is free.
   

5. The container also emits dependency mismatch warnings:

   Skipping import of cpp extensions due to incompatible torch version 2.11.0a0+eb65b36914.nv26.02 for torchao version 0.15.0
   

   transformers version 5.5.4 is incompatible with nvidia-modelopt and may cause issues.
   

additional notes

Additional notes

The main concern is that these appear to be documented/supported model families, but the official TensorRT-LLM container fails in multiple independent ways on NVIDIA L4:

• Gemma 3 PyTorch backend hits an internal metadata mismatch:

  TrtllmAttentionMetadata has no attribute kv_layout
  

• Gemma 4 E2B automatically selects FlashInfer:

  {'attn_backend': 'FLASHINFER'}
  

  but FlashInfer fails with:

  Unsupported architecture
  

• Gemma 3 AutoDeploy fails due to an apparent Transformers API mismatch:

  Gemma3ForConditionalGeneration.__init__() got an unexpected keyword argument 'use_cache'
  

• Gemma 4 E4B OOM may be expected on L4, but minimum memory requirements and multimodal/text-only behavior are unclear.

Questions for the TensorRT-LLM team:

1. Is google/gemma-3-4b-it expected to work with trtllm-serve --backend pytorch on NVIDIA L4 in release:1.3.0rc17?

2. Is google/gemma-4-E2B-it expected to work on NVIDIA L4?

3. Why does Gemma4ForConditionalGeneration automatically select FLASHINFER on L4 if the selected FlashInfer FMHA runner fails with Unsupported architecture?

4. Is there a supported way to force Gemma 3 / Gemma 4 to use a non-FlashInfer attention backend?

5. Is _autodeploy expected to support Gemma 3 in this container?

6. Should AutoDeploy pass use_cache into Gemma3ForConditionalGeneration.__init__()?

7. What is the recommended TensorRT-LLM container/tag for serving 4B-8B open-source LLMs on NVIDIA L4?

8. What are the minimum GPU memory requirements for Gemma 4 E2B and Gemma 4 E4B in TensorRT-LLM?

9. Can Gemma 4 multimodal models be served in text-only mode without loading the vision tower to GPU?

10. If these combinations are unsupported, can the supported-models documentation be updated with GPU/backend/memory-specific limitations?

Potential areas to investigate:

• FlashInfer TRTLLM FMHA runner support for SM89 / NVIDIA L4.
• TensorRT-LLM attention metadata compatibility with FlashInfer, specifically missing kv_layout.
• AutoDeploy compatibility with Transformers 5.5.4 and Gemma3ForConditionalGeneration.
• Whether Gemma 4 defaults should select FlashInfer only on supported GPU architectures.
• Whether the official container should include dependency versions that do not emit torchao/modelopt/transformers incompatibility warnings.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.