[Bug] awq for Qwen2-72B-instruct

### Checklist

- [X] 1. I have searched related issues but cannot get the expected help.
- [X] 2. The bug has not been fixed in the latest version.

### Describe the bug

针对qwen2-72b-instruct这个模型，使用awq进行量化，尝试了很多设置，都会出现如下bug error,麻烦尽快check下啊，是不是我的使用方式有什么问题呢？

Move model.norm to GPU.
Move lm_head to CPU.
Loading calibrate dataset ...
Using the latest cached version of the module from /hpc_stor01/home/guangfeng.liu/.cache/huggingface/modules/datasets_modules/datasets/ptb_text_only/8d1b97746fb9765d140e569ec5ddd35e20af4d37761f5e1bf357ea0b081f2c1f (last modified on Fri Jun 21 07:15:40 2024) since it couldn't be found locally at ptb_text_only, or remotely on the Hugging Face Hub.
Using the latest cached version of the module from /hpc_stor01/home/guangfeng.liu/.cache/huggingface/modules/datasets_modules/datasets/ptb_text_only/8d1b97746fb9765d140e569ec5ddd35e20af4d37761f5e1bf357ea0b081f2c1f (last modified on Fri Jun 21 07:15:40 2024) since it couldn't be found locally at ptb_text_only, or remotely on the Hugging Face Hub.
Token indices sequence length is longer than the specified maximum sequence length for this model (1104485 > 131072). Running this sequence through the model will result in indexing errors
model.layers.0, samples: 8, max gpu memory: 8.22 GB
Traceback (most recent call last):
File "/opt/py38/bin/lmdeploy", line 8, in
sys.exit(run())
File "/opt/py38/lib/python3.8/site-packages/lmdeploy/cli/entrypoint.py", line 37, in run
args.run(args)
File "/opt/py38/lib/python3.8/site-packages/lmdeploy/cli/lite.py", line 137, in auto_awq
auto_awq(**kwargs)
File "/opt/py38/lib/python3.8/site-packages/lmdeploy/lite/apis/auto_awq.py", line 96, in auto_awq
vl_model, model, tokenizer, work_dir = calibrate(model,
File "/opt/py38/lib/python3.8/site-packages/lmdeploy/lite/apis/calibrate.py", line 235, in calibrate
calib_ctx.calibrate(all_data)
File "/opt/py38/lib/python3.8/site-packages/lmdeploy/lite/quantization/calibration.py", line 315, in calibrate
_ = model(data.to(self.device))
File "/opt/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/py38/lib/python3.8/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 1034, in forward
layer_outputs = decoder_layer(
File "/opt/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/py38/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/opt/py38/lib/python3.8/site-packages/lmdeploy/lite/quantization/calibration.py", line 505, in _forward
auto_scale_block(mod, batch_kwargs[i], self.w_bits,
File "/opt/py38/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/opt/py38/lib/python3.8/site-packages/lmdeploy/lite/quantization/calibration.py", line 416, in auto_scale_block
_auto_get_scale(
File "/opt/py38/lib/python3.8/site-packages/lmdeploy/lite/quantization/calibration.py", line 400, in _auto_get_scale
best_ratio = _search_module_scale(module2inspect, layers, inp.value,
File "/opt/py38/lib/python3.8/site-packages/lmdeploy/lite/quantization/calibration.py", line 375, in _search_module_scale
fc.weight.data = pseudo_quantize_tensor(
File "/opt/py38/lib/python3.8/site-packages/lmdeploy/lite/quantization/awq.py", line 289, in pseudo_quantize_tensor
assert torch.isnan(scales).sum() == 0
AssertionError

### Reproduction

尝试了以下3个配置，均error
1.lmdeploy lite auto_awq ../pretrained-models/qwen2-72b-instruct/ --calib-dataset 'ptb' --calib-samples 128 --calib-seqlen 2048 --w-bits 4 --w-group-size 128 --work-dir ../pretrained-models/qwen2-72b-instruct-w4-lmdeploy-new2/ --batch-size 1
2.lmdeploy lite auto_awq ../pretrained-models/qwen2-72b-instruct/ --calib-dataset 'ptb' --calib-samples 32 --calib-seqlen 2048 --w-bits 4 --w-group-size 128 --work-dir ../pretrained-models/qwen2-72b-instruct-w4-lmdeploy-new2/ --batch-size 1
3.lmdeploy lite auto_awq ../pretrained-models/qwen2-72b-instruct/ --calib-dataset 'ptb' --calib-samples 32 --calib-seqlen 2048 --w-bits 4 --w-group-size 128 --work-dir ../pretrained-models/qwen2-72b-instruct-w4-lmdeploy-new2/ --batch-size 1 --search-scale True

### Environment

```Shell
都是在docker-v0.4.2版本操作
```

### Error traceback

_No response_