[Bug] latest benchmark run llama3.1-8b mat1 RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::Half

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/SpecForge/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

The error encountered is as follows:

[2025-12-03 10:45:41] Load weight end. type=LlamaForCausalLM, dtype=torch.bfloat16, avail mem=79.58 GB, mem usage=15.09 GB.
[2025-12-03 10:45:41] Using KV cache dtype: torch.bfloat16
[2025-12-03 10:45:42] KV Cache is allocated. #tokens: 264338, K size: 16.13 GB, V size: 16.13 GB
[2025-12-03 10:45:42] Memory pool end. avail mem=46.82 GB
[2025-12-03 10:45:42] Capture cuda graph begin. This can take up to several minutes. avail mem=46.27 GB
[2025-12-03 10:45:42] Capture cuda graph bs [1]
Capturing batches (bs=1 avail_mem=46.24 GB): 0%| | 0/1 [00:00<?, ?it/s][2025-12-03 10:45:42] MOE_A2A_BACKEND is not initialized, using default backend
Capturing batches (bs=1 avail_mem=46.24 GB): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 2.68it/s]
[2025-12-03 10:45:43] Capture cuda graph end. Time elapsed: 0.91 s. mem usage=-0.14 GB. avail mem=46.41 GB.
[2025-12-03 10:45:43] Warning: Target model's context_length (131072) is greater than the derived context_length (2048). This may lead to incorrect model outputs or CUDA errors. Note that the derived context_length may differ from max_position_embeddings in the model's config.
[2025-12-03 10:45:43] Overriding the draft model's max_position_embeddings to 131072.
[2025-12-03 10:45:43] Init torch distributed begin.
[2025-12-03 10:45:43] Init torch distributed ends. mem usage=0.00 GB
[2025-12-03 10:45:43] Load weight begin. avail mem=46.41 GB
Loading pt checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.96it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.96it/s]

[2025-12-03 10:45:44] Load weight end. type=LlamaForCausalLMEagle3, dtype=torch.float16, avail mem=44.20 GB, mem usage=2.21 GB.
[2025-12-03 10:45:44] Using KV cache dtype: torch.float16
[2025-12-03 10:45:44] KV Cache is allocated. #tokens: 264338, K size: 0.50 GB, V size: 0.50 GB
[2025-12-03 10:45:44] Memory pool end. avail mem=43.19 GB
[2025-12-03 10:45:45] Capture draft cuda graph begin. This can take up to several minutes. avail mem=44.21 GB
Capturing batches (bs=1 avail_mem=44.21 GB): 0%| | 0/1 [00:00<?, ?it/s]
[2025-12-03 10:45:47] Scheduler hit an exception: Traceback (most recent call last):
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py", line 133, in init
self.capture()
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py", line 161, in capture
CudaGraphRunner.capture(self)
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 497, in capture
) = self.capture_one_batch_size(bs, forward)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py", line 285, in capture_one_batch_size
run_once()
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py", line 273, in run_once
ret = self.eagle_worker.draft_forward(forward_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/speculative/eagle_worker.py", line 598, in draft_forward
logits_output, _ = self.draft_model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2209, in forward
output = self._forward_raw(
^^^^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2254, in _forward_raw
ret = self.forward_decode(
^^^^^^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2126, in forward_decode
return self.model.forward(
^^^^^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/models/llama.py", line 469, in forward
hidden_states = self.model(
^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/models/llama_eagle3.py", line 165, in forward
hidden_states, residual = self.midlayer(
^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/models/llama_eagle3.py", line 89, in forward
hidden_states = self.self_attn(
^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/models/llama.py", line 194, in forward
qkv, _ = self.qkv_proj(hidden_states)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/layers/linear.py", line 427, in forward
output_parallel = self.quant_method.apply(self, input, bias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/layers/quantization/unquant.py", line 130, in apply
return F.linear(x, layer.weight, bias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::Half

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2747, in run_scheduler_process
scheduler = Scheduler(
^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 323, in init
self.launch_draft_worker(
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 566, in launch_draft_worker
self.draft_worker = WorkerClass(
^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/speculative/eagle_worker.py", line 171, in init
self.init_cuda_graphs()
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/speculative/eagle_worker.py", line 213, in init_cuda_graphs
self.cuda_graph_runner = EAGLEDraftCudaGraphRunner(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/shenyuhao.syh/miniconda3/envs/eagle/lib/python3.11/site-packages/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py", line 135, in init
raise Exception(
Exception: Capture cuda graph failed: expected mat1 and mat2 to have the same dtype, but got: float != c10::Half
Possible solutions:

1. set --mem-fraction-static to a smaller value (e.g., 0.8 or 0.7)
2. set --cuda-graph-max-bs to a smaller value (e.g., 16)
3. disable torch compile by not using --enable-torch-compile
4. disable CUDA graph by --disable-cuda-graph. (Not recommended. Huge performance loss)
   Open an issue on GitHub https://github.com/sgl-project/sglang/issues/new/choose

Reproduction

The commands executed are as follows:

bash
base_PATH=meta-llama/Llama-3.1-8B-Instruct
Eagle3_PATH=/home/shenyuhao.syh/eagle3_llama8b_sg # Using the sglang version from lmsys
config_list=(
"1,5,8,32"
"2,5,8,32"
"4,3,1,4"
"8,3,1,4"
"1,0,0,0"
"2,0,0,0"
"4,0,0,0"
"8,0,0,0"
)

CUDA_VISIBLE_DEVICES=4 python3 bench_eagle3.py
--model-path $base_PATH
--speculative-draft-model-path $Eagle3_PATH
--port 30000
--mem-fraction-static 0.8
--tp-size 1
--config-list "${config_list[@]}"
--benchmark-list mtbench

Environment

Using the latest repository and UV environment.

Start with bench_eagle3.py and trace the reported failure through sglang/srt/speculative/eagle_draft_cuda_graph_runner.py, eagle_worker.py, and sglang/srt/models/llama_eagle3.py. Reproduce the Llama 3.1 8B Eagle3 benchmark using the provided command and configuration list, then identify the relevant dtype path. Done means the benchmark completes CUDA graph capture without the reported float-versus-half RuntimeError.