[REQUEST] ZeRO - introduce replicas to keep GBS from getting too large on hundres of gpus

Currently the GBS blows up to thousands if MBS is more than 1, which is counter-productive to training. And as clusters become larger and the training needs to happen faster this is becoming more and more of an issue.

e.g. take 512 gpus and MBS=16 - you end up with GBS of 8192 since GBS=MBS*N_GPUs

Possible solutions:

1. Repurpose Sequence Parallelism as Tensor Parallelism - so that the replica size is smaller
2. Introduce the concept of replicas - along the lines of ZeRO++ hybrid solution, except instead of taking advantage of local intra-node, it'd keep a replica to user's size - this of course will introduce an additional overhead of syncing replicas, but perhaps it could be mitigated by doing those syncs infrequently?

And surely one could think of other solution.

@tjruwase