[feature request] unable to override `dist.init_process_group` timeout in under `zero.Init`

When zero.Init is used how can a user override the timeout dist init arg? as the dist is inited here:

https://github.com/microsoft/DeepSpeed/blob/41a9bde14c808a75452baaa2609681316fc6912b/deepspeed/runtime/zero/partition_parameters.py#L654-L655

Of course the user could do something like this before instantiating the model:

from datetime import timedelta
import torch
[...]
                if not torch.distributed.is_initialized():
                    torch.distributed.init_process_group(backend="nccl", timeout=timedelta(seconds=3*60*60)))

but I'm not sure this would even work with deepspeed now using its own comms module and the init calls is much more complex - how would they know to provide the right args?

https://github.com/microsoft/DeepSpeed/blob/41a9bde14c808a75452baaa2609681316fc6912b/deepspeed/comm/comm.py#L376-L383

but perhaps the other alternative is to give users an API to override the default default_pg_timeout, so then they could just do:

form deepspeed import something
something.set_default_pg_timeout(3*60*60)
model = AutoModel.from_pretrained(...) # which internally calls `zero.Init` in transformers

for context: we are dealing with a silent crash in GPU followed by a timeout, so in order to be able to catch this event in action we were trying to extend the timeout to something much longer and run into this issue. Going to try the workaround I proposed at the top of this post.

@tjruwase