[Question]: CUDA out of memory

Describe the issue

Hi,
When I run test_llama.sh with llama3.1-8b-instruct, I meet Cuda out of memory. My gpu is L40 48G.
However, I can run test_minference_with_snapkv.sh without cuda out of memory.
I wonder if I want to run llama_test.sh with dense attention on single L40, do I have to truncate the length of the prompt until it can be run?
Or can I set multi-gpu parallel strategy? When I set CUDA_VISIBLE_DEVICES=0,1, the code running somewhere will report an error indicating that the two tensors are not on the same device.

python run_scbench.py
--task scbench_kv
--model_name_or_path /data/raid5/models/Llama-3.1-8B-Instruct
--data_dir ./data
--output_dir ./results
--rewrite
--attn_type dense
--kv_type dense
--use_chat_template
--trust_remote_code