Lightning-AI / Lightning-AI/pytorch-lightning
Enable parallel logging for private logging servers with high latency
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Description & Motivation
I am using a private MLFlow server, which has a very high latency. This results in my training process to be stuck in logging 99% of the time.
In my mind, this should be easy to fix - simply open a thread in parallel to the trainer which does the logging in the background, while the trainer keeps training.
### Pitch
Add a trainer flag "log_in_parallel" which detaches the logging from the training process, and lets it train before the logging is finished.
### Alternatives
An alternative would be to store all the logging data offline and only synchronize it in predetermined intervals, such as every 10 epochs.
### Additional context
99% is only a slight hyperbole, this is what my gpu usage looks like:

cc @borda
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file, test, or entry point is named in the issue. Start by locating the trainer's logging entry points and the existing private MLFlow-server integration; define how background logging, failures, ordering, shutdown, and the log_in_parallel flag should behave. Done means training continues without waiting for logging and the behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100