Dataloader shuffle_rng logic bug under multi-gpu settings?

It seems that each process under a multi-gpu train run uses a different seed for data shuffling (42 + process_rank as seen in dataloader.h line 172). This results in different random permutations of shard_indices as well as intra_shard_indices for processes and potentially leads to overlapping data load.

Is this expected? I would have expected random seeds for data shuffling to be the same for all processes.

Start at dataloader.h line 172 and trace how process_rank affects shuffle_rng for shard_indices and intra_shard_indices during multi-GPU training. Reproduce the behavior across processes and determine whether the differing permutations can cause overlapping data. Done means the behavior is confirmed as expected or the issue is narrowed to a validated correction.

Written by the indexing model from the issue text.