[Bug] 如何在指定卡上加载lmdeploy加速的模型

### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

当使用CUDA_VISIBLE_DEVICES指定可见卡不是0卡时，使用pipeline加载InternVL2_5-38B-MPO-AWQ模型会报错：
`ValueError: At least one of the model submodule will be offloaded to disk, please pass along an `offload_folder`.`
debug发现部分vit的权重是放在cpu上的，尝试直接使用to方法迁到gpu上失败。请问这个问题该怎么解决

### Reproduction

pipeline(model_path, backend_config=TurbomindEngineConfig(session_len=10000),)

### Environment

```Shell
InternVL2_5-38B-MPO-AWQ
```

### Error traceback

```Shell

```