intel / intel/auto-round

[Bug]: Kimi-K2.6_mxfp4_fp8attn_LLMC vllm eval qkv scale is not loaded

Open
#2,370 1 comment 0 reactions 1 assignee Claimed by @yiliu30 View on GitHub
bug
Dominant language
Python
Stars
1.6k
Forks
175
Avg merge
1d 18h
Merged PRs (30d)
99

Description

### Problem Description

Kimi-K2.6_mxfp4_fp8attn_LLMC quantized model run vllm eval can't load qkv scale
https://inteltf-jenk.sh.intel.com/job/Intel-GPU-key-model-test/1152/artifact/evaluation_Kimi-K2.6_mxfp4.log

### Reproduction Steps

cd /software/data/jenkins/CS09_B200/workspace/Intel-GPU-key-model-test/neural-compressor/examples/pytorch/nlp/huggingface_models/language-modeling/quantization/auto_round/kimi
bash run_benchmark.sh --model_path=/home/hshen/suyue/saved_models/Kimi-K2.6_mxfp4_fp8attn_LLMC/Kimi-K2.6-mxfp-w4g32 --tasks=hellaswag,piqa,mmlu,gsm8k --static_attention_dtype=fp8

### Environment Information

https://inteltf-jenk.sh.intel.com/job/Intel-GPU-key-model-test/1152/artifact/evaluation_Kimi-K2.6_mxfp4.log

### Error Logs

```shell
(Worker_TP0 pid=7609) WARNING 09-11 08:05:24 [weight_utils.py:1462] Found k_scale in the checkpoint (e.g. layers.3.self_attn.attn.k_scale), but not found the expected name in the model (e.g. layers.3.self_attn.attn.attn.k_scale). k_scale is not loaded.
(Worker_TP0 pid=7609) WARNING 09-11 08:05:24 [weight_utils.py:1462] Found q_scale in the checkpoint (e.g. layers.3.self_attn.attn.q_scale), but not found the expected name in the model (e.g. layers.3.self_attn.attn.attn.q_scale). q_scale is not loaded.
(Worker_TP0 pid=7609) WARNING 09-11 08:05:24 [weight_utils.py:1462] Found v_scale in the checkpoint (e.g. layers.3.self_attn.attn.v_scale), but not found the expected name in the model (e.g. layers.3.self_attn.attn.attn.v_scale). v_scale is not loaded.
```

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.