llamav2 13b multi-gpu issue

Hi all, I am using vLLM worker to serve up llama v2 13b model. Please check my engine args below

2023-08-01 00:41:32 | INFO | __main__ | Using engine args:
2023-08-01 00:41:32 | INFO | __main__ | engine_args.model: /opt/resources/model/XX/
2023-08-01 00:41:32 | INFO | __main__ | engine_args.tokenizer: /opt/resources/model/XX/
2023-08-01 00:41:32 | INFO | __main__ | engine_args.tokenizer_mode: slow
2023-08-01 00:41:32 | INFO | __main__ | engine_args.download_dir: None
2023-08-01 00:41:32 | INFO | __main__ | engine_args.use_np_weights: False
2023-08-01 00:41:32 | INFO | __main__ | engine_args.use_dummy_weights: False
2023-08-01 00:41:32 | INFO | __main__ | engine_args.dtype: auto
2023-08-01 00:41:32 | INFO | __main__ | engine_args.seed: 0
2023-08-01 00:41:32 | INFO | __main__ | engine_args.worker_use_ray: True
2023-08-01 00:41:32 | INFO | __main__ | engine_args.pipeline_parallel_size: 1
2023-08-01 00:41:32 | INFO | __main__ | engine_args.tensor_parallel_size: 4
2023-08-01 00:41:32 | INFO | __main__ | engine_args.block_size: 16
2023-08-01 00:41:32 | INFO | __main__ | engine_args.swap_space: 4
2023-08-01 00:41:32 | INFO | __main__ | engine_args.gpu_memory_utilization: 0.95
2023-08-01 00:41:32 | INFO | __main__ | engine_args.max_num_batched_tokens: 1800
2023-08-01 00:41:32 | INFO | __main__ | engine_args.max_num_seqs: 256
2023-08-01 00:41:32 | INFO | __main__ | engine_args.disable_log_stats: False
2023-08-01 00:41:32 | INFO | __main__ | engine_args.engine_use_ray: False
2023-08-01 00:41:32 | INFO | __main__ | engine_args.disable_log_requests: False
2023-08-01 00:41:32 | INFO | __main__ | Loading model /opt/resources/model/XX/

When I try to run using a single/dual (with tensor parallel) A100, it works just fine. However, when I try to use 2 or 4 A10Gs the model never ends up getting served/loaded. My nvidia-smi mem usage looks like below going from 0MiB to 2MiB and model doesn't load beyond that.

Start by reproducing the vLLM worker launch with the shown engine arguments, especially tensor_parallel_size 2 or 4, on A10G GPUs, and compare it with the working A100 setup. Inspect the model-loading stage after the “Loading model” message and collect the relevant worker or runtime logs. Done means the Llama 2 13B model loads and becomes served on the reported A10G configurations.