Lightning-AI / Lightning-AI/pytorch-lightning
PyTorch Lightning: Slower Training Speed with CSVLogger, Can Asynchronous Logging be Configured?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Description & Motivation
I am currently using PyTorch Lightning for training, and I'm using the CSVLogger callback to record the metric values during training to a CSV file. However, I've noticed that enabling CSVLogger and starting the training process significantly slows down the overall training speed.
I suspect that this slowdown might be caused by the default synchronous writing of metric values to the CSV file by CSVLogger, which could introduce delays due to IO operations. In order to optimize the training speed, I would like to know if there is a way to configure CSVLogger in PyTorch Lightning to perform asynchronous logging.
Therefore, I'm interested in learning how to configure CSVLogger to perform asynchronous logging, which could potentially improve the training speed.
If anyone has any experience or suggestions on how to configure CSVLogger for asynchronous logging in PyTorch Lightning, I would greatly appreciate it. Thank you very much!
### Pitch
_No response_
### Alternatives
_No response_
### Additional context
_No response_
cc @lantiga @borda
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the CSVLogger entry point and reviewing how it writes metric values during training. Determine whether asynchronous logging can be configured, and validate that the change avoids the reported slowdown without losing metric records.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100