Feature request: different batch size per GPU

Assume you have two GPUs: 1x 3090 24GB and 1x 3060 12GB.
When using DDP or DeepSpeed, the 3090 will not be able to use more than 12GB and will not be fully utilized because of this.

Some solutions are discussed for DDP in the following issue (see "Alternatives"): https://github.com/pytorch/pytorch/issues/67253
Another solution is using a custom DistributedSampler: https://discuss.pytorch.org/t/can-ddp-divide-dataset-unevenly-to-workers/146381/4

In constrast to DDP, DeepSpeed does not support this at all at the moment as it seems.

Start by examining the project's DDP and DeepSpeed integration, then read the linked PyTorch issue and custom DistributedSampler discussion. The issue names no files or tests, so first identify the relevant training entry points and existing batch-size handling. Done would mean supporting different per-GPU batch sizes in DeepSpeed while preserving correct distributed training.

Written by the indexing model from the issue text.