DistributedManager initialization fails on CPU-only hosts

Version

2.1.1 (also reproduces on current main)

On which installation method(s) does this occur?

Pip

Describe the issue

DistributedManager cannot initialize on CPU-only hosts (e.g. laptop-scale gloo runs, CPU CI):

1. DistributedManager.setup() passes manager.device as the device_id argument of torch.distributed.init_process_group. On CPU-only hosts that is torch.device("cpu"), which recent PyTorch versions reject with ValueError: init_process_group device_id parameter must be an accelerator with an index.
2. When local_rank is None, setup() computes rank % torch.cuda.device_count(), a division by zero when no accelerator is present (hit via the SLURM/OpenMPI initialization paths, which pass local_rank=None).
3. As a follow-on, the failed initialization leaves _shared_state["_distributed"] = True, so the atexit cleanup() calls dist.destroy_process_group() on a process group that was never created and dies with AssertionError: Process group cannot be None, obscuring the original error.

Note the distributed manager tests are skipped on CPU by an autouse fixture, so CI does not catch this path.

Minimum reproducible example

# repro.py — run on a CPU-only machine:
#   torchrun --standalone --nproc_per_node=2 repro.py
import torch
from physicsnemo.distributed import DistributedManager

DistributedManager.initialize()
dm = DistributedManager()
t = torch.tensor([dm.rank + 1.0])
torch.distributed.broadcast(t, src=0)
DistributedManager.cleanup()

Relevant log output

  File ".../physicsnemo/distributed/manager.py", line 590, in setup
    dist.init_process_group(
  File ".../torch/distributed/distributed_c10d.py", line 1763, in init_process_group
    raise ValueError(
ValueError: init_process_group device_id parameter must be an accelerator with an index

Exception ignored in atexit callback: <staticmethod(<function DistributedManager.cleanup ...)>
  File ".../physicsnemo/distributed/manager.py", line 823, in cleanup
    dist.destroy_process_group()
AssertionError: Process group cannot be None

Environment details

nvidia-physicsnemo 2.1.1, torch 2.13.0, Python 3.12, macOS (arm64) and Linux CPU nodes; gloo backend.

Start in physicsnemo/distributed/manager.py around setup() near line 590 and cleanup() near line 823. Review the CPU-skipping distributed-manager test fixture and the provided torchrun reproduction, then exercise the SLURM/OpenMPI paths with the gloo backend. Done means CPU-only initialization works without division-by-zero or invalid device arguments, failed setup does not leave misleading cleanup errors, and tests cover the path.