[Bug] baichuan2 7b-13b cannot be 8-bit weight quantized with different error stack.

### Checklist

- [ ] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.

### Describe the bug

baichuan2 7b-13b cannot be 8-bit weight quantized.
for baichuan2 7b, the error is: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 40.00 GiB. GPU 0 has a total capacty of 79.21 GiB of which 32.42 GiB is free.
for baichuan2 13b, the error is: RuntimeError: Currently, quantification and calibration of BaichuanForCausalLM are not supported. The supported model types are InternLMForCausalLM, InternLM2ForCausalLM, QWenLMHeadModel, BaiChuanForCausalLM, LlamaForCausalLM.

### Reproduction

```
lmdeploy lite smooth_quant $path_to_Baichuan2-13B-Chat --work-dir /nvme/qa_test_models/Baichuan2-13B-Chat-inner-w8a8

lmdeploy lite smooth_quant $path_to_Baichuan2-7B-Chat --work-dir /nvme/qa_test_models/Baichuan2-7B-Chat-inner-w8a8
```

### Environment

```Shell
sys.platform: linux
Python: 3.10.0 (default, Mar 3 2022, 09:58:08) [GCC 7.5.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0,1,2,3,4,5,6,7: NVIDIA A100-SXM4-80GB
CUDA_HOME: /usr/local/cuda-11.7
NVCC: Cuda compilation tools, release 11.7, V11.7.64
GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
PyTorch: 2.1.0+cu118
PyTorch compiling details: PyTorch built with:
- GCC 9.3
- C++ Version: 201703
- Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v3.1.1 (Git Hash 64f6bcbcbab628e96f33a62c3e975f8535a7bde4)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX512
- CUDA Runtime 11.8
- NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_90,code=sm_90
- CuDNN 8.7
- Magma 2.6.1
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-invalid-partial-specialization -Wno-unused-private-field -Wno-aligned-allocation-unavailable -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.1.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

TorchVision: 0.16.0+cu118
LMDeploy: 0.2.3+c13f840
transformers: 4.33.0
gradio: 3.50.2
fastapi: 0.109.2
pydantic: 2.6.1
```

### Error traceback

```Shell
for Baichuan2-13B-Chatm, the error is:
/home/zhulin1/miniconda3/envs/lmdeployv23/lib/python3.10/site-packages/fuzzywuzzy/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers
pip install xformers.
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:19<00:00, 6.56s/it]
Move model to GPU.
Move lm_head to CPU.
Loading calibrate dataset ...
Token indices sequence length is longer than the specified maximum sequence length for this model (1138791 > 4096). Running this sequence through the model will result in indexing errors
Traceback (most recent call last):
File "/home/zhulin1/miniconda3/envs/lmdeployv23/bin/lmdeploy", line 8, in
sys.exit(run())
File "/home/zhulin1/miniconda3/envs/lmdeployv23/lib/python3.10/site-packages/lmdeploy/cli/entrypoint.py", line 18, in run
args.run(args)
File "/home/zhulin1/miniconda3/envs/lmdeployv23/lib/python3.10/site-packages/lmdeploy/cli/lite.py", line 152, in smooth_quant
smooth_quant(**kwargs)
File "/home/zhulin1/miniconda3/envs/lmdeployv23/lib/python3.10/site-packages/lmdeploy/lite/apis/smooth_quant.py", line 74, in smooth_quant
model, tokenizer, work_dir = calibrate(model, calib_dataset, calib_samples,
File "/home/zhulin1/miniconda3/envs/lmdeployv23/lib/python3.10/site-packages/lmdeploy/lite/apis/calibrate.py", line 193, in calibrate
calib_ctx.calibrate(all_data)
File "/home/zhulin1/miniconda3/envs/lmdeployv23/lib/python3.10/site-packages/lmdeploy/lite/quantization/calibration.py", line 310, in calibrate
_ = model(data.to(self.device))
File "/home/zhulin1/miniconda3/envs/lmdeployv23/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/zhulin1/miniconda3/envs/lmdeployv23/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/zhulin1/.cache/huggingface/modules/transformers_modules/Baichuan2-13B-Chat/modeling_baichuan.py", line 467, in forward
layer_outputs = decoder_layer(
File "/home/zhulin1/miniconda3/envs/lmdeployv23/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/zhulin1/miniconda3/envs/lmdeployv23/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/zhulin1/.cache/huggingface/modules/transformers_modules/Baichuan2-13B-Chat/modeling_baichuan.py", line 244, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "/home/zhulin1/miniconda3/envs/lmdeployv23/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/zhulin1/miniconda3/envs/lmdeployv23/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/zhulin1/.cache/huggingface/modules/transformers_modules/Baichuan2-13B-Chat/modeling_baichuan.py", line 186, in forward
attn_weights = torch.matmul(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 40.00 GiB. GPU 0 has a total capacty of 79.21 GiB of which 32.42 GiB is free. Including non-PyTorch memory, this process has 46.79 GiB memory in use. Of the allocated memory 43.48 GiB is allocated by PyTorch, and 2.51 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

for Baichuan2-7B-Chat the error is:

/home/zhulin1/miniconda3/envs/lmdeployv23/lib/python3.10/site-packages/fuzzywuzzy/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers
pip install xformers.
Move model.embed_tokens to GPU.
Move model.layers.0 to CPU.
Move model.layers.1 to CPU.
Move model.layers.2 to CPU.
Move model.layers.3 to CPU.
Move model.layers.4 to CPU.
Move model.layers.5 to CPU.
Move model.layers.6 to CPU.
Move model.layers.7 to CPU.
Move model.layers.8 to CPU.
Move model.layers.9 to CPU.
Move model.layers.10 to CPU.
Move model.layers.11 to CPU.
Move model.layers.12 to CPU.
Move model.layers.13 to CPU.
Move model.layers.14 to CPU.
Move model.layers.15 to CPU.
Move model.layers.16 to CPU.
Move model.layers.17 to CPU.
Move model.layers.18 to CPU.
Move model.layers.19 to CPU.
Move model.layers.20 to CPU.
Move model.layers.21 to CPU.
Move model.layers.22 to CPU.
Move model.layers.23 to CPU.
Move model.layers.24 to CPU.
Move model.layers.25 to CPU.
Move model.layers.26 to CPU.
Move model.layers.27 to CPU.
Move model.layers.28 to CPU.
Move model.layers.29 to CPU.
Move model.layers.30 to CPU.
Move model.layers.31 to CPU.
Move model.norm to GPU.
Move lm_head to CPU.
Loading calibrate dataset ...
Token indices sequence length is longer than the specified maximum sequence length for this model (1138791 > 4096). Running this sequence through the model will result in indexing errors
model.layers.0, samples: 128, max gpu memory: 8.35 GB
model.layers.1, samples: 128, max gpu memory: 10.35 GB
model.layers.2, samples: 128, max gpu memory: 10.36 GB
model.layers.3, samples: 128, max gpu memory: 10.36 GB
model.layers.4, samples: 128, max gpu memory: 10.37 GB
model.layers.5, samples: 128, max gpu memory: 10.37 GB
model.layers.6, samples: 128, max gpu memory: 10.37 GB
model.layers.7, samples: 128, max gpu memory: 10.38 GB
model.layers.8, samples: 128, max gpu memory: 10.38 GB
model.layers.9, samples: 128, max gpu memory: 10.38 GB
model.layers.10, samples: 128, max gpu memory: 10.39 GB
model.layers.11, samples: 128, max gpu memory: 10.39 GB
model.layers.12, samples: 128, max gpu memory: 10.40 GB
model.layers.13, samples: 128, max gpu memory: 10.40 GB
model.layers.14, samples: 128, max gpu memory: 10.40 GB
model.layers.15, samples: 128, max gpu memory: 10.41 GB
model.layers.16, samples: 128, max gpu memory: 10.41 GB
model.layers.17, samples: 128, max gpu memory: 10.42 GB
model.layers.18, samples: 128, max gpu memory: 10.42 GB
model.layers.19, samples: 128, max gpu memory: 10.42 GB
model.layers.20, samples: 128, max gpu memory: 10.43 GB
model.layers.21, samples: 128, max gpu memory: 10.43 GB
model.layers.22, samples: 128, max gpu memory: 10.44 GB
model.layers.23, samples: 128, max gpu memory: 10.44 GB
model.layers.24, samples: 128, max gpu memory: 10.44 GB
model.layers.25, samples: 128, max gpu memory: 10.45 GB
model.layers.26, samples: 128, max gpu memory: 10.45 GB
model.layers.27, samples: 128, max gpu memory: 10.46 GB
model.layers.28, samples: 128, max gpu memory: 10.46 GB
model.layers.29, samples: 128, max gpu memory: 10.46 GB
model.layers.30, samples: 128, max gpu memory: 10.47 GB
model.layers.31, samples: 128, max gpu memory: 10.47 GB
Traceback (most recent call last):
File "/home/zhulin1/miniconda3/envs/lmdeployv23/bin/lmdeploy", line 8, in
sys.exit(run())
File "/home/zhulin1/miniconda3/envs/lmdeployv23/lib/python3.10/site-packages/lmdeploy/cli/entrypoint.py", line 18, in run
args.run(args)
File "/home/zhulin1/miniconda3/envs/lmdeployv23/lib/python3.10/site-packages/lmdeploy/cli/lite.py", line 152, in smooth_quant
smooth_quant(**kwargs)
File "/home/zhulin1/miniconda3/envs/lmdeployv23/lib/python3.10/site-packages/lmdeploy/lite/apis/smooth_quant.py", line 84, in smooth_quant
raise RuntimeError(
RuntimeError: Currently, quantification and calibration of BaichuanForCausalLM are not supported. The supported model types are InternLMForCausalLM, InternLM2ForCausalLM, QWenLMHeadModel, BaiChuanForCausalLM, LlamaForCausalLM.
```