Lightning-AI / Lightning-AI/pytorch-lightning
Inconsistent version number with DDP due to bad setup (2.0.6)
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
+ The bug is introduced by 2.0.5 or 2.0.6, where different ranks of DDP may get inconsistent version number of a training instance.
+ Reproduce: set breakpoint to `def version(self) of CSVLogger`, to check the first time when `version()` is called/
+ Figures: first version 2.0.4, second 2.0.6
+ Issue: the `setup` method is not called in 2.0.6, so that different DDP ranks may get different version numbers
+ To the maintainer: you are making too many changes between minor version fixes
Best
### What version are you seeing the problem on?
v2.0
### How to reproduce the bug
_No response_
### Error messages and logs
_No response_
### Environment
_No response_
### More info
_No response_
cc @lantiga @borda
Contributor guide
Research direction
Start at CSVLogger.version and its setup method, using the reported v2.0 DDP behavior and breakpoint as the initial reproduction path. Compare the first and subsequent version values across ranks; done means setup is consistently involved and DDP ranks no longer receive inconsistent version numbers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100