[Usage]: Qwen + Eagle3 supported in PyTorch backend yet?

System Info

System Information:

• OS: using the nvcr.io/nvidia/tensorrt-llm/release:1.0.0rc5 docker image
• Python version: 3.12.3
• CUDA version: 12.9
• GPU model(s): 8*H100
• Driver version: 535.161.08
• TensorRT-LLM version: 1.0.0rc5

How would you like to use TensorRT-LLM

I am trying to run a custom trained Qwen2.5 model with Eagle3 speculative decoding quantized into FP8 on TensorRT-LLM PyTorch backend, however, as of version 1.0.0rc5, I see that is it possibly not supported yet.

Could you please clarify whether it is possible at the moment, because the support matrix suggests that Eagle3 is implemented already (specifically, Eagle3, judging by the presence of the corresponding config), and also it was possible to train and quantize the model with Eagle3 trained with your example code using the examples from speculative_decoding and llm_ptq directories, however when trying to initialize the tensorrt_llm.LLM with the following arguments

{
  "max_seq_len": 32768, 
  "max_num_tokens": 32768, 
  "max_batch_size": 128, 
  "tensor_parallel_size": 2, 
  "disable_overlap_scheduler": false, 
  "enable_chunked_prefill": true, 
  "enable_attention_dp": false, 
  "enable_trtllm_sampler": false, 
  "print_iter_log": false, 
  "enable_iter_perf_stats": false
  "load_format": "auto", 
  "build_config": {
    "max_input_len": 32768,
    "max_seq_len": 32768, 
    "gather_generation_logits": true, 
    "kv_cache_type": "paged", 
    "speculative_decoding_mode": "eagle", 
    "plugin_config": {
      "dtype": "float16", 
      "tokens_per_block": 32, 
      "multiple_profiles": true, 
      "paged_state": true, 
      "remove_input_padding": true, 
      "use_paged_context_fmha": true, 
      "use_fp8_context_fmha": true, 
      "use_fused_mlp": true
    }, 
  }, 
  "attn_backend": "TRTLLM", 
  "pipeline_parallel_size": 1, 
  "enable_lora": false, 
  "kv_cache_config": {
    "dtype": "auto", 
    "enable_block_reuse": false, 
    "free_gpu_memory_fraction": 0.95
  }, 
  "scheduler_config": {
    "context_chunking_policy": "FIRST_COME_FIRST_SERVED", 
    "capacity_scheduler_policy": "GUARANTEED_NO_EVICT"
  }, 
}

An error ValueError: Unknown architecture for AutoModelForCausalLM: EagleQwen2ForCausalLM is returned

If I try to add EagleQwen2ForCausalLM to the registry myself by copying and pasting the Llama implementation and substituting LlamaConfig with Qwen2Config I encounter numerous errors related to required Eagle weights not being found, such as

[TRT-LLM] [E] Error executing load_single_module with args ('model.midlayer.hidden_norm', RMSNorm()): 'weight'

And entries added to the model.safetensors.index.json after training the speculative decoding model look like this

"eagle_module.fc.bias": "model-00007-of-00008.safetensors",
"eagle_module.fc.input_scale": "model-00007-of-00008.safetensors",
"eagle_module.fc.weight": "model-00007-of-00008.safetensors",
"eagle_module.fc.weight_scale": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.input_layernorm.weight": "model-00008-of-00008.safetensors",
"eagle_module.layers.0.mlp.down_proj.input_scale": "model-00008-of-00008.safetensors",
"eagle_module.layers.0.mlp.down_proj.weight": "model-00008-of-00008.safetensors",
"eagle_module.layers.0.mlp.down_proj.weight_scale": "model-00008-of-00008.safetensors",
"eagle_module.layers.0.mlp.gate_proj.input_scale": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.mlp.gate_proj.weight": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.mlp.gate_proj.weight_scale": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.mlp.up_proj.input_scale": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.mlp.up_proj.weight": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.mlp.up_proj.weight_scale": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.post_attention_layernorm.weight": "model-00008-of-00008.safetensors",
"eagle_module.layers.0.self_attn.k_proj.bias": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.self_attn.k_proj.input_scale": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.self_attn.k_proj.weight": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.self_attn.k_proj.weight_scale": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.self_attn.o_proj.input_scale": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.self_attn.o_proj.weight": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.self_attn.o_proj.weight_scale": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.self_attn.q_proj.bias": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.self_attn.q_proj.input_scale": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.self_attn.q_proj.weight": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.self_attn.q_proj.weight_scale": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.self_attn.v_proj.bias": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.self_attn.v_proj.input_scale": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.self_attn.v_proj.weight": "model-00007-of-00008.safetensors",
"eagle_module.layers.0.self_attn.v_proj.weight_scale": "model-00007-of-00008.safetensors",
"eagle_module.norm.weight": "model-00008-of-00008.safetensors",

Can these weights be somehow converted into ones the backend will recognize? Or should I use different tools for the task I am pursuing?

I am using editable versions of tensorrt_llm==1.0.0rc5 and nvidia-modelopt==0.33.0 with no changes to the code related to this issue.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Start at the AutoModelForCausalLM architecture registry and modeling_speculative.py around the Llama implementation, then compare them with the ModelOpt Eagle3 configuration and the listed safetensors names. Determine whether EagleQwen2ForCausalLM needs backend support or a weight conversion path; done means a documented, working loading path or a clear statement that the combination is unsupported.