GPTQ 4bit Error RuntimeError: Error(s) in loading state_dict for LlamaForCausalLM: 2024-02-12 21:24:19 | ERROR | stderr | Unexpected key(s) in state_dict:

Nobody has claimed this yet.

I followed the instructions in gptq.md. And there is an error:
Loading model ...
2024-02-12 21:24:19 | ERROR | stderr | Traceback (most recent call last):
2024-02-12 21:24:19 | ERROR | stderr | File "/home/kyrie/miniconda3/envs/llm/lib/python3.10/runpy.py", line 196, in _run_module_as_main
2024-02-12 21:24:19 | ERROR | stderr | return _run_code(code, main_globals, None,
2024-02-12 21:24:19 | ERROR | stderr | File "/home/kyrie/miniconda3/envs/llm/lib/python3.10/runpy.py", line 86, in _run_code
2024-02-12 21:24:19 | ERROR | stderr | exec(code, run_globals)
2024-02-12 21:24:19 | ERROR | stderr | File "/home/kyrie/miniconda3/envs/llm/lib/python3.10/site-packages/fastchat/serve/model_worker.py", line 375, in
2024-02-12 21:24:19 | ERROR | stderr | args, worker = create_model_worker()
2024-02-12 21:24:19 | ERROR | stderr | File "/home/kyrie/miniconda3/envs/llm/lib/python3.10/site-packages/fastchat/serve/model_worker.py", line 346, in create_model_worker
2024-02-12 21:24:19 | ERROR | stderr | worker = ModelWorker(
2024-02-12 21:24:19 | ERROR | stderr | File "/home/kyrie/miniconda3/envs/llm/lib/python3.10/site-packages/fastchat/serve/model_worker.py", line 77, in __init__
2024-02-12 21:24:19 | ERROR | stderr | self.model, self.tokenizer = load_model(
2024-02-12 21:24:19 | ERROR | stderr | File "/home/kyrie/miniconda3/envs/llm/lib/python3.10/site-packages/fastchat/model/model_adapter.py", line 310, in load_model
2024-02-12 21:24:19 | ERROR | stderr | model, tokenizer = load_gptq_quantized(model_path, gptq_config)
2024-02-12 21:24:19 | ERROR | stderr | File "/home/kyrie/miniconda3/envs/llm/lib/python3.10/site-packages/fastchat/modules/gptq.py", line 55, in load_gptq_quantized
2024-02-12 21:24:19 | ERROR | stderr | model = load_quant(
2024-02-12 21:24:19 | ERROR | stderr | File "/home/kyrie/llm/GPTQ-for-LLaMa/llama.py", line 310, in load_quant
2024-02-12 21:24:19 | ERROR | stderr | model.load_state_dict(torch.load(checkpoint))
2024-02-12 21:24:19 | ERROR | stderr | File "/home/kyrie/miniconda3/envs/llm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2152, in load_state_dict
2024-02-12 21:24:19 | ERROR | stderr | raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
2024-02-12 21:24:19 | ERROR | stderr | RuntimeError: Error(s) in loading state_dict for LlamaForCausalLM:
2024-02-12 21:24:19 | ERROR | stderr | Unexpected key(s) in state_dict: "model.layers.0.self_attn.rotary_emb.inv_freq", "model.layers.1.self_attn.rotary_emb.inv_freq", "model.layers.2.self_attn.rotary_emb.inv_freq", "model.layers.3.self_attn.rotary_emb.inv_freq", "model.layers.4.self_attn.rotary_emb.inv_freq", "model.layers.5.self_attn.rotary_emb.inv_freq", "model.layers.6.self_attn.rotary_emb.inv_freq", "model.layers.7.self_attn.rotary_emb.inv_freq", "model.layers.8.self_attn.rotary_emb.inv_freq", "model.layers.9.self_attn.rotary_emb.inv_freq", "model.layers.10.self_attn.rotary_emb.inv_freq", "model.layers.11.self_attn.rotary_emb.inv_freq", "model.layers.12.self_attn.rotary_emb.inv_freq", "model.layers.13.self_attn.rotary_emb.inv_freq", "model.layers.14.self_attn.rotary_emb.inv_freq", "model.layers.15.self_attn.rotary_emb.inv_freq", "model.layers.16.self_attn.rotary_emb.inv_freq", "model.layers.17.self_attn.rotary_emb.inv_freq", "model.layers.18.self_attn.rotary_emb.inv_freq", "model.layers.19.self_attn.rotary_emb.inv_freq", "model.layers.20.self_attn.rotary_emb.inv_freq", "model.layers.21.self_attn.rotary_emb.inv_freq", "model.layers.22.self_attn.rotary_emb.inv_freq", "model.layers.23.self_attn.rotary_emb.inv_freq", "model.layers.24.self_attn.rotary_emb.inv_freq", "model.layers.25.self_attn.rotary_emb.inv_freq", "model.layers.26.self_attn.rotary_emb.inv_freq", "model.layers.27.self_attn.rotary_emb.inv_freq", "model.layers.28.self_attn.rotary_emb.inv_freq", "model.layers.29.self_attn.rotary_emb.inv_freq", "model.layers.30.self_attn.rotary_emb.inv_freq", "model.layers.31.self_attn.rotary_emb.inv_freq".

Looking forward to your reply ASAP.

Start with gptq.md and reproduce the command using the reported environment. Then trace loading through fastchat/modules/gptq.py, fastchat/model/model_adapter.py, and GPTQ-for-LLaMa/llama.py, focusing on the checkpoint passed to load_state_dict. Done means the documented GPTQ model loads without the reported unexpected rotary-embedding keys.

Written by the indexing model from the issue text.