[Help] device map question of runing on multi card (2 x RTX3090)

Nobody has claimed this yet.

the model of Vicuna-13B need 26G Vram, while 1 x RTX3090 has only 24G of Vram, so I am trying a way to run it on 2 x RTX3090. But I find it very slow when using the default mode, even slower than 1 x RTX3090 in 8-bit mode. I don't got NVLink, so I think it may be the Vram transfer speed limit.

I want to change the device_map to try if I can get a bit faster, what is the best practise to set the device map so that I can run it a bit faster with 2 x RTX3090? I am using v0 model, and my currently setting is like this:

device_map = {
'model.embed_tokens': 0,
'model.layers.0': 0,
'model.layers.1': 0,
'model.layers.2': 0,
'model.layers.3': 0,
'model.layers.4': 0,
'model.layers.5': 0,
'model.layers.6': 0,
'model.layers.7': 0,
'model.layers.8': 0,
'model.layers.9': 0,
'model.layers.10': 0,
'model.layers.11': 0,
'model.layers.12': 0,
'model.layers.13': 0,
'model.layers.14': 0,
'model.layers.15': 0,
'model.layers.16': 0,
'model.layers.17': 0,
'model.layers.18': 0,
'model.layers.19': 0,
'model.layers.20': 0,
'model.layers.21': 1,
'model.layers.22': 1,
'model.layers.23': 1,
'model.layers.24': 1,
'model.layers.25': 1,
'model.layers.26': 1,
'model.layers.27': 1,
'model.layers.28': 1,
'model.layers.29': 1,
'model.layers.30': 1,
'model.layers.31': 1,
'model.layers.32': 1,
'model.layers.33': 1,
'model.layers.34': 1,
'model.layers.35': 1,
'model.layers.36': 1,
'model.layers.37.self_attn': 1,
'model.layers.37.mlp.gate_proj': 1,
'model.layers.37.mlp.down_proj': 1,
'model.layers.37.mlp.up_proj': 1,
'model.layers.37.mlp.act_fn': 1,
'model.layers.37.input_layernorm': 1,
'model.layers.37.post_attention_layernorm': 1,
'model.layers.38': 1,
'model.layers.39': 1,
'model.norm': 1,
'lm_head': 1
}

Start by reproducing the Vicuna-13B multi-GPU run with the supplied device_map and compare it with the reported single-card 8-bit performance. Done would require identifying a device mapping that improves speed on two RTX3090 cards and documenting the measured result.

Written by the indexing model from the issue text.