Lightning-AI / Lightning-AI/pytorch-lightning

Non-reproducible results with num_workers=0

Open
#20,679 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug reproducibility ver: 2.5.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

Hello,
I have a question related to obtaining reproducible results when setting `num_workers` in a `torch.DataLoader` and using `pl.LightningDataModule` and `pl.Trainer`.

So far I am experiencing the following: when i set `num_workers=0` the results are different to those obtained when `num_workers>0` for any epoch other than the first (and my understanding is that the way the data is reshuffled within each epoch is somehow the same for all `num_workers>0` but different for `num_workers=0`). Is this an expected behaviour? Any suggestion to prevent it?

If you think this is a bug, I can provide a concise example that reproduces what I see (which however requires stripping out code from an internal library so I'd rather do it only if you think what I get is likely a bug and not an expected behaviour 😄 )

Thanks!

### What version are you seeing the problem on?

v2.5

### How to reproduce the bug

See code in the comment below

### Error messages and logs

No real error, just results are not reproducible for different `num_workers`

### Environment

Current environment

```
#- PyTorch Lightning Version (e.g., 2.5.0): 2.5.0
#- PyTorch Version (e.g., 2.5): 2.6.0
#- Python version (e.g., 3.12): 3.12.2
#- OS (e.g., Linux): Linux
#- CUDA/cuDNN version: running on CPU
#- GPU models and configuration: -
#- How you installed Lightning(`conda`, `pip`, source): conda
```

### More info

_No response_

cc @lantiga

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproduction code referenced in the comment and review how torch.DataLoader, pl.LightningDataModule, and pl.Trainer handle shuffling and worker counts. Compare results across epochs for num_workers=0 and num_workers>0, then establish whether the difference is expected or identify the behavior that needs correction.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.