[Bug]: AutoDeploy Gemma E2B/E4B fails with special_tokens AttributeError during initialization

System Info

• Container: container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc12
• HW: x86_64 + RTX Pro 6000
• Model: google/gemma-4-E2B-it and google/gemma-4-E4B-it

Who can help?

@suyoggupta @bmarimuthu-nv

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

(Generally following the Gemma 4 cookbook)

1. trtllm-serve "google/gemma-4-E2B-it" \
     --host 0.0.0.0 \
     --port 8000 \
     --backend _autodeploy \
     --trust_remote_code \
     --extra_llm_api_options examples/auto_deploy/model_registry/configs/gemma3n_e2b_it.yaml
   

2. The server process will fail to initialize the tokenizer.

Expected behavior

The server should load the checkpoint and start correctly.

actual behavior

The server process crashes while loading the model/tokenizer with:

Traceback (most recent call last):
  File "/usr/local/bin/trtllm-serve", line 6, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1406, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/commands/serve.py", line 994, in serve
    _serve_llm()
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/commands/serve.py", line 964, in _serve_llm
    launch_server(host,
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/commands/serve.py", line 306, in launch_server
    llm = AutoDeployLLM(**llm_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/auto_deploy/llm.py", line 133, in __init__
    super().__init__(*args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 1239, in __init__
    super().__init__(model,
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 252, in __init__
    self._build_model()
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/auto_deploy/llm.py", line 167, in _build_model
    super()._build_model()
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 1289, in _build_model
    self._tokenizer = self._try_load_tokenizer()
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/auto_deploy/llm.py", line 139, in _try_load_tokenizer
    return tokenizer_factory(self.factory.init_tokenizer())
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/auto_deploy/models/hf.py", line 334, in init_tokenizer
    return AutoTokenizer.from_pretrained(self.tokenizer, **self.tokenizer_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/tokenization_auto.py", line 1156, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 2113, in from_pretrained
    return cls._from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 2359, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma/tokenization_gemma_fast.py", line 100, in __init__
    super().__init__(
  File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_fast.py", line 178, in __init__
    super().__init__(**kwargs)
  File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 1472, in __init__
    self._set_model_specific_special_tokens(special_tokens=self.extra_special_tokens)
  File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 1210, in _set_model_specific_special_tokens
    self.SPECIAL_TOKENS_ATTRIBUTES = self.SPECIAL_TOKENS_ATTRIBUTES + list(special_tokens.keys())
                                                                           ^^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'keys'

additional notes

AD seems to be sensitive to the location of the model_factory property in the configuration file. For Gemma 4 26B/31B, this has to be the first property in the configuration file to apply correctly. However, applying this change to the example configuration yields the same error (AutoModelForCausalLM is likely the default).

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.