Unable to Load Llama-2-70B-chat-GPTQ

Nobody has claimed this yet.

I'm trying to run this model: https://huggingface.co/TheBloke/Llama-2-70B-chat-GPTQ through this documentation: https://github.com/lm-sys/FastChat/blob/main/docs/gptq.md.

I'm running this command:
`python3 -m fastchat.serve.cli --model-path ../../llm/tmp/Llama-2-70B-chat-GPTQ/ --gptq-wbits 4 --gptq-groupsize -32 `

However, I'm getting this error with the branch: [gptq-4bit-32g-actorder_True](https://huggingface.co/TheBloke/Llama-2-70B-chat-GPTQ/tree/gptq-4bit-32g-actorder_True)

Loading GPTQ quantized model...
Traceback (most recent call last):
File "/home/ianwu/anaconda3/envs/fastchat/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/ianwu/anaconda3/envs/fastchat/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/ianwu/avatar/FastChat/fastchat/serve/cli.py", line 280, in
main(args)
File "/home/ianwu/avatar/FastChat/fastchat/serve/cli.py", line 206, in main
chat_loop(
File "/home/ianwu/avatar/FastChat/fastchat/serve/inference.py", line 302, in chat_loop
model, tokenizer = load_model(
File "/home/ianwu/avatar/FastChat/fastchat/model/model_adapter.py", line 252, in load_model
model, tokenizer = load_gptq_quantized(model_path, gptq_config)
File "/home/ianwu/avatar/FastChat/fastchat/modules/gptq.py", line 55, in load_gptq_quantized
model = load_quant(
File "/home/ianwu/avatar/FastChat/fastchat/../repositories/GPTQ-for-LLaMa/llama.py", line 301, in load_quant
quant.make_quant_linear(model, layers, wbits, groupsize, act_order)
File "/home/ianwu/avatar/FastChat/fastchat/../repositories/GPTQ-for-LLaMa/quant/quant_linear.py", line 401, in make_quant_linear
make_quant_linear(child, names, bits, groupsize,act_order, name + '.' + name1 if name != '' else name1)
File "/home/ianwu/avatar/FastChat/fastchat/../repositories/GPTQ-for-LLaMa/quant/quant_linear.py", line 401, in make_quant_linear
make_quant_linear(child, names, bits, groupsize,act_order, name + '.' + name1 if name != '' else name1)
File "/home/ianwu/avatar/FastChat/fastchat/../repositories/GPTQ-for-LLaMa/quant/quant_linear.py", line 401, in make_quant_linear
make_quant_linear(child, names, bits, groupsize,act_order, name + '.' + name1 if name != '' else name1)
[Previous line repeated 1 more time]
File "/home/ianwu/avatar/FastChat/fastchat/../repositories/GPTQ-for-LLaMa/quant/quant_linear.py", line 399, in make_quant_linear
setattr(module, attr, QuantLinear(bits, groupsize, tmp.in_features, tmp.out_features, tmp.bias is not None, act_order=act_order))
File "/home/ianwu/avatar/FastChat/fastchat/../repositories/GPTQ-for-LLaMa/quant/quant_linear.py", line 315, in __init__
self.register_buffer('qzeros', torch.zeros((math.ceil(infeatures / self.groupsize), outfeatures // 32 * self.bits), dtype=torch.int32))
RuntimeError: Trying to create tensor with negative dimension -256: [-256, 128]

Start with docs/gptq.md and the command shown, then trace loading through fastchat/modules/gptq.py into repositories/GPTQ-for-LLaMa/llama.py and quant/quant_linear.py. Check how the supplied GPTQ branch and arguments reach QuantLinear, and confirm done when the Llama-2-70B-chat-GPTQ model loads without the negative-dimension error.

Written by the indexing model from the issue text.