GPTQ error

Open

#2,893 opened on Jan 7, 2024 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language: Python
Stars: 39.5k
Forks: 4.8k
PR merge metrics: No merged PRs in 30d

Description

I followed the instruction in gptq.md, and met the following error:
2024-01-07 20:47:51 | ERROR | stderr | Traceback (most recent call last):
2024-01-07 20:47:51 | ERROR | stderr | File "/home/kyrie/miniconda3/envs/py39/lib/python3.9/runpy.py", line 197, in _run_module_as_main
2024-01-07 20:47:51 | ERROR | stderr | return _run_code(code, main_globals, None,
2024-01-07 20:47:51 | ERROR | stderr | File "/home/kyrie/miniconda3/envs/py39/lib/python3.9/runpy.py", line 87, in _run_code
2024-01-07 20:47:51 | ERROR | stderr | exec(code, run_globals)
2024-01-07 20:47:51 | ERROR | stderr | File "/home/kyrie/miniconda3/envs/py39/lib/python3.9/site-packages/fastchat/serve/model_worker.py", line 369, in
2024-01-07 20:47:51 | ERROR | stderr | args, worker = create_model_worker()
2024-01-07 20:47:51 | ERROR | stderr | File "/home/kyrie/miniconda3/envs/py39/lib/python3.9/site-packages/fastchat/serve/model_worker.py", line 341, in create_model_worker
2024-01-07 20:47:51 | ERROR | stderr | worker = ModelWorker(
2024-01-07 20:47:51 | ERROR | stderr | File "/home/kyrie/miniconda3/envs/py39/lib/python3.9/site-packages/fastchat/serve/model_worker.py", line 77, in __init__
2024-01-07 20:47:51 | ERROR | stderr | self.model, self.tokenizer = load_model(
2024-01-07 20:47:51 | ERROR | stderr | File "/home/kyrie/miniconda3/envs/py39/lib/python3.9/site-packages/fastchat/model/model_adapter.py", line 298, in load_model
2024-01-07 20:47:51 | ERROR | stderr | model, tokenizer = load_gptq_quantized(model_path, gptq_config)
2024-01-07 20:47:51 | ERROR | stderr | File "/home/kyrie/miniconda3/envs/py39/lib/python3.9/site-packages/fastchat/modules/gptq.py", line 55, in load_gptq_quantized
2024-01-07 20:47:51 | ERROR | stderr | model = load_quant(
2024-01-07 20:47:51 | ERROR | stderr | File "/home/kyrie/GPTQ-for-LLaMa/llama.py", line 310, in load_quant
2024-01-07 20:47:51 | ERROR | stderr | model.load_state_dict(torch.load(checkpoint))
2024-01-07 20:47:51 | ERROR | stderr | File "/home/kyrie/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 2152, in load_state_dict
2024-01-07 20:47:51 | ERROR | stderr | raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
2024-01-07 20:47:51 | ERROR | stderr | RuntimeError: Error(s) in loading state_dict for LlamaForCausalLM:
2024-01-07 20:47:51 | ERROR | stderr | Unexpected key(s) in state_dict: "model.layers.0.self_attn.rotary_emb.inv_freq", "model.layers.1.self_attn.rotary_emb.inv_freq", "model.layers.2.self_attn.rotary_emb.inv_freq", "model.layers.3.self_attn.rotary_emb.inv_freq", "model.layers.4.self_attn.rotary_emb.inv_freq", "model.layers.5.self_attn.rotary_emb.inv_freq", "model.layers.6.self_attn.rotary_emb.inv_freq", "model.layers.7.self_attn.rotary_emb.inv_freq", "model.layers.8.self_attn.rotary_emb.inv_freq", "model.layers.9.self_attn.rotary_emb.inv_freq", "model.layers.10.self_attn.rotary_emb.inv_freq", "model.layers.11.self_attn.rotary_emb.inv_freq", "model.layers.12.self_attn.rotary_emb.inv_freq", "model.layers.13.self_attn.rotary_emb.inv_freq", "model.layers.14.self_attn.rotary_emb.inv_freq", "model.layers.15.self_attn.rotary_emb.inv_freq", "model.layers.16.self_attn.rotary_emb.inv_freq", "model.layers.17.self_attn.rotary_emb.inv_freq", "model.layers.18.self_attn.rotary_emb.inv_freq", "model.layers.19.self_attn.rotary_emb.inv_freq", "model.layers.20.self_attn.rotary_emb.inv_freq", "model.layers.21.self_attn.rotary_emb.inv_freq", "model.layers.22.self_attn.rotary_emb.inv_freq", "model.layers.23.self_attn.rotary_emb.inv_freq", "model.layers.24.self_attn.rotary_emb.inv_freq", "model.layers.25.self_attn.rotary_emb.inv_freq", "model.layers.26.self_attn.rotary_emb.inv_freq", "model.layers.27.self_attn.rotary_emb.inv_freq", "model.layers.28.self_attn.rotary_emb.inv_freq", "model.layers.29.self_attn.rotary_emb.inv_freq", "model.layers.30.self_attn.rotary_emb.inv_freq", "model.layers.31.self_attn.rotary_emb.inv_freq".

Contributor guide

No contributing guide indexed for this repository

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Research direction

Start with gptq.md and the load_gptq_quantized entry point in fastchat/modules/gptq.py, then inspect the reported load_quant call in GPTQ-for-LLaMa/llama.py. Reproduce the command with the same checkpoint and compare the model state keys involved in the failure. Done should include a confirmed compatibility fix or a documented setup correction with a regression check.

Written by the indexing model from the issue text.

Assessment

Tech stack: python, pytorch
Domain: backend, machine-learning
Issue type: Bug
Difficulty: 4/5
Estimated time: 3-5 days
Activity status: Stale
Clarity: Needs clarification
Newbie friendliness: 25/100