Lightning-AI / Lightning-AI/pytorch-lightning
Race condition between `AsyncCheckpointIO` and `WandbLogger`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
When `AsyncCheckpointIO` is used and `WandbLogger` is configured to log model artifacts, there is a race condition that can occur. The notion of race conditions with `AsyncCheckpointIO` is already documented in #14035, but I feel that the topic of race conditions specifically involving `WandbLogger` may be distinct enough to warrant its own issue.
In short, the race condition happens when `WandbLogger` tries to log a checkpoint artifact that doesn't exist yet because `AsyncCheckpointIO` hasn't finished writing it.
I wouldn't know how to implement a thread lock or future callback in `WandbLogger`, but it would be nice if there's an easy solution here.
### What version are you seeing the problem on?
v2.0
### How to reproduce the bug
_No response_
### Error messages and logs
```
# Error messages and logs here please
```
### Environment
Current environment
```
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):
```
### More info
_No response_
cc @lantiga @morganmcg1 @borisdayma @scottire @parambharat
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the interaction between AsyncCheckpointIO and WandbLogger, focusing on when checkpoint writing completes and when artifact logging begins. Reproduce the reported race if possible; done means WandbLogger does not attempt to log a checkpoint artifact before AsyncCheckpointIO has finished writing it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100