[launcher] optionally log exceptions only on node 0 process 0

Currently, when working on a multi-node setup, exceptions are hard to figured out as they are logged simultaneously by many processes (e.g. 64!) and the bt is interleaved which makes it very difficult to figure out what went wrong. Usually it's the same same exception (usually OOM).

pytorch launcher has the same issue.

So one of the possibilities is to optionally /dev/null stderr on all processes, but node 0 process 0.

Most likely at
https://github.com/microsoft/DeepSpeed/blob/a029239812e15cf35334514449ed3127b915780a/deepspeed/launcher/launch.py#L148

The local and global rank info here: https://github.com/microsoft/DeepSpeed/blob/a029239812e15cf35334514449ed3127b915780a/deepspeed/launcher/launch.py#L117-L118

The condition is then to /dev/null on current_env["RANK"] > 0.

or potentially replicate the same approach as torch elastic launcher torch.distributed.run does:

  -r REDIRECTS, --redirects REDIRECTS
                        Redirect std streams into a log file in the log
                        directory (e.g. [-r 3] redirects both stdout+stderr
                        for all workers, [-r 0:1,1:2] redirects stdout for
                        local rank 0 and stderr for local rank 1).
  -t TEE, --tee TEE     Tee std streams into a log file and also to console
                        (see --redirects for format).

but this one doesn't handle global rank, so asking first at https://github.com/pytorch/pytorch/issues/60911

Start in deepspeed/launcher/launch.py around lines 117-148 and inspect how local and global ranks and stderr are handled. Compare the requested node 0/process 0 filtering with the torch.distributed.run redirect and tee options. Done means an optional launcher behavior prevents interleaved exception output from other processes while preserving useful output from node 0 process 0.