[Bug] AWQ Model Fails Loading ADapter

### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.

### Describe the bug

When running the repo example I choose:
`YurtsAI/Meta-Llama-3-8B-Instruct-AWQ` model
and
`traderpedroso/llama3-8b-lora` this adapter.

I know the adapter was trained on the 4bit base model. Im not sure if this works with awq

``` File "/home/merlin/code/kreacher/venv/lib/python3.10/site-packages/lmdeploy/serve/async_engine.py", line 252, in _build_pytorch
self.engine = Engine(model_path=model_path,
File "/home/merlin/code/kreacher/venv/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 153, in __init__
_paging_adapters(adapters,
File "/home/merlin/code/kreacher/venv/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine.py", line 68, in _paging_adapters
model_agent.paging_adapters(weight_maps)
File "/home/merlin/code/kreacher/venv/lib/python3.10/site-packages/lmdeploy/pytorch/engine/model_agent.py", line 715, in paging_adapters
weight_map.cache_adapter(lora_linears, cpu_caches)
File "/home/merlin/code/kreacher/venv/lib/python3.10/site-packages/lmdeploy/pytorch/adapter/adapter.py", line 226, in cache_adapter
assert len(lora_linears) == len(caches), (
AssertionError: len(lora_linears) == len(caches)
```

If I comment out `len(lora_linears) == len(caches)` then the adapter merges... but im not sure if that its supposed to work like that or not.

### Reproduction

My script:
```python
from lmdeploy import pipeline, GenerationConfig, PytorchEngineConfig

backend_config = PytorchEngineConfig(session_len=2048,
adapters=dict(lora_name_1='traderpedroso/llama3-8b-lora'))
gen_config = GenerationConfig(top_p=0.8,
top_k=40,
temperature=0.8,
max_new_tokens=1024)
pipe = pipeline('YurtsAI/Meta-Llama-3-8B-Instruct-AWQ',
backend_config=backend_config)
prompts = [[{
'role': 'user',
'content': '您猜怎么着'
}]]
response = pipe(prompts, gen_config=gen_config, adapter_name='lora_name_1')
print(response)
```

### Environment

```Shell
Running latest version of LMDeploy.
```

### Error traceback

_No response_