[Bug] 使用领域数据sft qwen2-7b后，转awq 报错

### Checklist

- [ ] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.

### Describe the bug

使用lmdeploy lite auto_awq将sft后的qwen2-7b进行awq量化，报错assert torch.isnan(p).sum() == 0

### Reproduction

lmdeploy lite auto_awq \
qwen2-sft-checkpoint-1506-merged \
--calib-dataset 'c4' \
--calib-samples 128 \
--calib-seqlen 4096 \
--work-dir qwen2_7b_qg_2_epoch_awq

### Environment

```Shell
lmdeploy==0.4.1
```

### Error traceback

```Shell
Traceback (most recent call last):
File "/opt/conda/bin/lmdeploy", line 8, in
sys.exit(run())
File "/opt/conda/lib/python3.8/site-packages/lmdeploy/cli/entrypoint.py", line 37, in run
args.run(args)
File "/opt/conda/lib/python3.8/site-packages/lmdeploy/cli/lite.py", line 131, in auto_awq
auto_awq(**kwargs)
File "/opt/conda/lib/python3.8/site-packages/lmdeploy/lite/apis/auto_awq.py", line 68, in auto_awq
smooth_layers(layers, fc2fcs, norm2fcs, act_scales, w_group_size, device)
File "/opt/conda/lib/python3.8/site-packages/lmdeploy/lite/quantization/awq.py", line 242, in smooth_layers
smooth_ln_fcs(ln, fcs, a_scales[a_name], group_size)
File "/opt/conda/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/lmdeploy/lite/quantization/awq.py", line 118, in smooth_ln_fcs
assert torch.isnan(p).sum() == 0
AssertionError
```