How to train the model on 4 GPUs?

I am finetuning the model with my own dataset, but having an error with only 4 GPU. I was able to finetune the model with 8 GPUs but keeping getting errors when train on 4 GPUs, even though I have set the batch sizes to the following:

ARG_WORLD_SIZE=${1:-1}
ARG_NPROC_PER_NODE=${2:-4}

GLOBAL_BATCH_SIZE=4
LOCAL_BATCH_SIZE=1
GRADIENT_ACCUMULATION_STEPS=$[$GLOBAL_BATCH_SIZE/($WORLD_SIZE*$NPROC_PER_NODE*$LOCAL_BATCH_SIZE)]

The command i used: CUDA_VISIBLE_DEVICES=0, 1,2,3 ./va_joint_custorm.sh

Errors:

```
Traceback (most recent call last):
File "/home/ubuntu/qixuewei/Projects/VideoLLaMA2/videollama2/train.py", line 683, in
train()
File "/home/ubuntu/qixuewei/Projects/VideoLLaMA2/videollama2/train.py", line 664, in train
trainer.train(resume_from_checkpoint=True)
File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 1932, in train
return inner_training_loop(
File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 2113, in _inner_training_loop
deepspeed_load_checkpoint(
File "/opt/conda/lib/python3.10/site-packages/transformers/integrations/deepspeed.py", line 433, in deepspeed_load_checkpoint
load_path, _ = deepspeed_engine.load_checkpoint(
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2751, in load_checkpoint
success = self._load_zero_checkpoint(load_dir, tag, load_optimizer_states=load_optimizer_states)
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2932, in _load_zero_checkpoint
raise ZeRORuntimeException("The checkpoint being loaded used a DP " \
deepspeed.runtime.zero.utils.ZeRORuntimeException: The checkpoint being loaded used a DP world size of 8 but the current world size is 4. Automatic adjustment of ZeRO's optimizer state partitioning with a new world size is not currently supported.
Traceback (most recent call last):
File "/home/ubuntu/qixuewei/Projects/VideoLLaMA2/videollama2/train.py", line 683, in
train()
File "/home/ubuntu/qixuewei/Projects/VideoLLaMA2/videollama2/train.py", line 664, in train
trainer.train(resume_from_checkpoint=True)
File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 1932, in train
return inner_training_loop(
File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 2113, in _inner_training_loop
deepspeed_load_checkpoint(
File "/opt/conda/lib/python3.10/site-packages/transformers/integrations/deepspeed.py", line 433, in deepspeed_load_checkpoint
load_path, _ = deepspeed_engine.load_checkpoint(
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2751, in load_checkpoint
success = self._load_zero_checkpoint(load_dir, tag, load_optimizer_states=load_optimizer_states)
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2932, in _load_zero_checkpoint
raise ZeRORuntimeException("The checkpoint being loaded used a DP " \
deepspeed.runtime.zero.utils.ZeRORuntimeException: The checkpoint being loaded used a DP world size of 8 but the current world size is 4. Automatic adjustment of ZeRO's optimizer state partitioning with a new world size is not currently supported.
Traceback (most recent call last):
File "/home/ubuntu/qixuewei/Projects/VideoLLaMA2/videollama2/train.py", line 683, in
train()
File "/home/ubuntu/qixuewei/Projects/VideoLLaMA2/videollama2/train.py", line 664, in train
trainer.train(resume_from_checkpoint=True)
File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 1932, in train
return inner_training_loop(
File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 2113, in _inner_training_loop
deepspeed_load_checkpoint(
File "/opt/conda/lib/python3.10/site-packages/transformers/integrations/deepspeed.py", line 433, in deepspeed_load_checkpoint
load_path, _ = deepspeed_engine.load_checkpoint(
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2751, in load_checkpoint
success = self._load_zero_checkpoint(load_dir, tag, load_optimizer_states=load_optimizer_states)
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2932, in _load_zero_checkpoint
raise ZeRORuntimeException("The checkpoint being loaded used a DP " \ deepspeed.runtime.zero.utils.ZeRORuntimeException: The checkpoint being loaded used a DP world size of 8 but the current world size is 4. Automatic adjustment of ZeRO's optimizer state partitioning with a new world size is not currently supported.
[2025-03-01 03:07:40,362] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 1618129) of binary: /opt/conda/bin/python3.10
Traceback (most recent call last):
File "/opt/conda/bin/torchrun", line 8, in
sys.exit(main())
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
return f(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main
run(args)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run
elastic_launch(
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
videollama2/train.py FAILED
------------------------------------------------------------
Failures:
[1]:
time : 2025-03-01_03:07:40
host : ip-172-31-7-155.eu-north-1.compute.internal
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 1618130)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2025-03-01_03:07:40
host : ip-172-31-7-155.eu-north-1.compute.internal
rank : 2 (local_rank: 2)
exitcode : 1 (pid: 1618131)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
time : 2025-03-01_03:07:40
host : ip-172-31-7-155.eu-north-1.compute.internal
rank : 3 (local_rank: 3)
exitcode : 1 (pid: 1618132)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2025-03-01_03:07:40
host : ip-172-31-7-155.eu-north-1.compute.internal
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 1618129)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
```

Start with videollama2/train.py around the call to trainer.train(resume_from_checkpoint=True), then inspect the va_joint_custorm.sh entry point and the checkpoint configuration used by the torchrun command. Reproduce the reported resume attempt with four GPUs and compare it with the eight-GPU run; done means the four-GPU training path is either supported or its limitation is clearly documented.

Written by the indexing model from the issue text.