Lightning-AI / Lightning-AI/pytorch-lightning

Inconsistent version number with DDP due to bad setup (2.0.6)

Open
#18,294 0 comments 0 reactions 0 assignees View on GitHub
bug logger logger: csv ver: 2.0.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

+ The bug is introduced by 2.0.5 or 2.0.6, where different ranks of DDP may get inconsistent version number of a training instance.
+ Reproduce: set breakpoint to `def version(self) of CSVLogger`, to check the first time when `version()` is called/
+ Figures: first version 2.0.4, second 2.0.6
+ Issue: the `setup` method is not called in 2.0.6, so that different DDP ranks may get different version numbers

2-0-4
2-0-6

+ To the maintainer: you are making too many changes between minor version fixes

Best

### What version are you seeing the problem on?

v2.0

### How to reproduce the bug

_No response_

### Error messages and logs

_No response_

### Environment

_No response_

### More info

_No response_

cc @lantiga @borda

Contributor guide

Open the contributing guide

Research direction

Start at CSVLogger.version and its setup method, using the reported v2.0 DDP behavior and breakpoint as the initial reproduction path. Compare the first and subsequent version values across ranks; done means setup is consistently involved and DDP ranks no longer receive inconsistent version numbers.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.