S-LoRA with pytorch backend is very slow

### Motivation

I using the scripts to **benchmark s-lora** with lmdeploy 0.2.6 on 2*A30. Firstly I only benchmark the base model lama2-13b-hf, the performance of pytorch backend **is obviously lower** than using turbormind backend, **It is almost unusable**.

Besides, if the `tp=2`, the utilization of gpu 0 is almost 0 but the gpu1 is always 100%, why?

```bash
# benchmark/benchmark_pytorch_engine_a100.sh
# https://github.com/InternLM/lmdeploy/blob/main/benchmark/profile_throughput.py
python3 profile_throughput.py data.csv /data/models/Llama-2-13b-hf/ \
--backend pytorch --tp 2 --concurrency 1 --cache-max-entry-count 0.5 \
--request_out_len 256 --top-k 3 --top-p 0.95 --temperature 0.0 --repetition-penalty 1.15 \
--num-prompts 10 --csv perf.csv

```



### Related resources

_No response_

### Additional context

_No response_