Lightning-AI / Lightning-AI/pytorch-lightning

Enable parallel logging for private logging servers with high latency

Open
#18,967 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

discussion feature logger: mlflow
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Description & Motivation

I am using a private MLFlow server, which has a very high latency. This results in my training process to be stuck in logging 99% of the time.

In my mind, this should be easy to fix - simply open a thread in parallel to the trainer which does the logging in the background, while the trainer keeps training.

### Pitch

Add a trainer flag "log_in_parallel" which detaches the logging from the training process, and lets it train before the logging is finished.

### Alternatives

An alternative would be to store all the logging data offline and only synchronize it in predetermined intervals, such as every 10 epochs.

### Additional context

99% is only a slight hyperbole, this is what my gpu usage looks like:
![image](https://github.com/Lightning-AI/lightning/assets/11627602/3ff758e8-89a4-4b22-8408-45e445ccf9e2)

cc @borda

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No file, test, or entry point is named in the issue. Start by locating the trainer's logging entry points and the existing private MLFlow-server integration; define how background logging, failures, ordering, shutdown, and the log_in_parallel flag should behave. Done means training continues without waiting for logging and the behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.