Failed running AWQ 4bit example

Nobody has claimed this yet.

I strictly followed the steps listed in this tutorial https://github.com/lm-sys/FastChat/blob/main/docs/awq.md. However, met the following error. Any idea from experienced users?

system info:
- A100 GPU
- FastChat 0.2.33
- awq 0.1.0
- torch 2.0.1
- CUDA 11.7

```
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for LlamaForCausalLM:
Unexpected key(s) in state_dict: "model.layers.0.self_attn.rotary_emb.inv_freq", "model.layers.1.self_attn
.rotary_emb.inv_freq", "model.layers.2.self_attn.rotary_emb.inv_freq", "model.layers.3.self_attn.rotary_emb.inv_fr
eq", "model.layers.4.self_attn.rotary_emb.inv_freq", "model.layers.5.self_attn.rotary_emb.inv_freq", "model.layers
.6.self_attn.rotary_emb.inv_freq", "model.layers.7.self_attn.rotary_emb.inv_freq", "model.layers.8.self_attn.rotar
y_emb.inv_freq", "model.layers.9.self_attn.rotary_emb.inv_freq", "model.layers.10.self_attn.rotary_emb.inv_freq",
"model.layers.11.self_attn.rotary_emb.inv_freq", "model.layers.12.self_attn.rotary_emb.inv_freq", "model.layers.13
.self_attn.rotary_emb.inv_freq", "model.layers.14.self_attn.rotary_emb.inv_freq", "model.layers.15.self_attn.rotar
y_emb.inv_freq", "model.layers.16.self_attn.rotary_emb.inv_freq", "model.layers.17.self_attn.rotary_emb.inv_freq",
"model.layers.18.self_attn.rotary_emb.inv_freq", "model.layers.19.self_attn.rotary_emb.inv_freq", "model.layers.2
0.self_attn.rotary_emb.inv_freq", "model.layers.21.self_attn.rotary_emb.inv_freq", "model.layers.22.self_attn.rota
ry_emb.inv_freq", "model.layers.23.self_attn.rotary_emb.inv_freq", "model.layers.24.self_attn.rotary_emb.inv_freq"
, "model.layers.25.self_attn.rotary_emb.inv_freq", "model.layers.26.self_attn.rotary_emb.inv_freq", "model.layers.
27.self_attn.rotary_emb.inv_freq", "model.layers.28.self_attn.rotary_emb.inv_freq", "model.layers.29.self_attn.rot
ary_emb.inv_freq", "model.layers.30.self_attn.rotary_emb.inv_freq", "model.layers.31.self_attn.rotary_emb.inv_freq
".
size mismatch for model.layers.0.self_attn.k_proj.qzeros: copying a param with shape torch.Size([32, 512])
from checkpoint, the shape in current model is torch.Size([4096, 4]). ... more
```

Start with docs/awq.md and reproduce the AWQ 4-bit example using the reported FastChat, awq, PyTorch, CUDA, and A100 versions. Compare the checkpoint and model-loading steps against the state_dict errors, then verify that the example loads successfully without unexpected keys or tensor shape mismatches.

Written by the indexing model from the issue text.