2 node speed is not faster than 1 node

I use 1 node with 4V100，got 700it/s，and 1 node with 4P40，got 300it/s，
but when I use 2 nodes with 4V100 and 4P40 by deepspeed，got
“4 pytorch allocator cache flushes since last step. this happens when there is high memory pressure and is detrimental to performance. if this is happening frequently consider adjusting settings to reduce memory consumption. If you are unable to make the cache flushes go away consider adding get_accelerator().empty_cache() calls in your training loop to ensure that all ranks flush their caches at the same time”
this is my script

`
#!/bin/bash
export NCCL_IB_DISABLE=1;
export NCCL_P2P_DISABLE=1;
export NCCL_DEBUG=INFO;
export NCCL_SOCKET_IFNAME=enp,ens;
export CXX=g++;
deepspeed --hostfile hostfile
--master_addr p40
--master_port 29600
fastchat/train/train.py
--model_name_or_path /data2/lhw/FastChat/models/vicuna-7b
--data_path /data2/lhw/FastChat/playground/data/onlineque_v2.0.json
--fp16 True
--output_dir ./output
--num_train_epochs 3
--per_device_train_batch_size 1
--per_device_eval_batch_size 1
--gradient_accumulation_steps 32
--evaluation_strategy "no"
--save_strategy "steps"
--save_steps 1200
--save_total_limit 10
--learning_rate 2e-5
--weight_decay 0.
--warmup_ratio 0.03
--lr_scheduler_type "cosine"
--logging_steps 1
--tf32 False
--model_max_length 2048
--gradient_checkpointing True
--lazy_preprocess True
--deepspeed /data2/lhw/FastChat/fastchat/train/deepspeed-config.json

`

and deep speed-config.json is

json { "zero_optimization": { "stage": 3, "offload_optimizer": { "device": "cpu", "pin_memory": true }, "overlap_comm": true, "contiguous_gradients": true }, "optimizer": { "type": "AdamW", "params": { "lr": "auto", "betas": "auto", "eps": "auto", "weight_decay": "auto" } }, "train_micro_batch_size_per_gpu": "auto" }
I use train.py because of flash_attn is not support v100 and p40.

Start with fastchat/train/train.py and the referenced deepspeed-config.json; reproduce the one-node and two-node runs using the supplied launch flags and training settings. Done means the multi-node throughput regression and allocator-cache-flush warning are explained, with the cause and any applicable project or configuration change documented.