Lightning-AI / Lightning-AI/pytorch-lightning
Inconsistent usage of paths in the experimental loggers
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Outline & Motivation
The usage of [`log_dir`](https://github.com/Lightning-AI/pytorch-lightning/blob/9177ec09caadcf88859e1f1e3e10a18e8832069a/src/lightning/fabric/loggers/logger.py#L47) and [`save_dir`](https://github.com/Lightning-AI/pytorch-lightning/blob/9177ec09caadcf88859e1f1e3e10a18e8832069a/src/lightning/pytorch/loggers/logger.py#L45) are confusion.
Imaging we are implement a new logger, let's see how we can define the `save_dir` property.
```python
class CustomLogger(Logger):
def __init__(self, all_experiment_path):
self._root_dir = all_experiment_path
@property
@override
def save_dir(self) -> str:
option1 = os.path.join(self._root_dir, self.name, str(self.version))
# option2 = self._root_dir
# option3 = os.path.join(self._root_dir, self.name)
return option1 # or option2 or option3
```
The first option includes the experiment name and version as subdirectories, and the second one does not.
As `CustomLogger` is not a subclass of `TensorBoardLogger` or `CSVLogger`. the `Trainer.log_dir` is defined to be `logger.save_dir` if the first logger is a `CustomLogger`, according to [here](https://github.com/Lightning-AI/pytorch-lightning/blob/9177ec09caadcf88859e1f1e3e10a18e8832069a/src/lightning/pytorch/cli.py#L252C13-L252C20).
The `config.yaml` file will be saved to `logger.save_dir`/`config.yaml`. The checkpoint file will be saved to `logger.save_dir`/`logger.name`/`logger.version`, which
expand to be `logger._root_dir`/`logger.name`/`logger.version`/`logger.name`/`logger.version`, which is with duplicate directory hierarchy, according to [`__resolve_ckpt_dir()`](https://github.com/Lightning-AI/pytorch-lightning/blob/9177ec09caadcf88859e1f1e3e10a18e8832069a/src/lightning/pytorch/callbacks/model_checkpoint.py#L610) function.
I **DO NOT** like it. The following tree view shows this clearly.
```
└── experiment_name
├── version_0
│ ├── config.yaml
│ └── experiment_name
│ └── version_0
│ └── checkpoints
│ └── epoch=7-step=3000.ckpt
└── version_1
├── config.yaml
└── experiment_name
└── version_1
└── checkpoints
└── epoch=28-step=10875.ckpt
```
However, there is no way to remove the duplicate hierarchy for now. If the second or third option is adopted, the `config.yaml` file will be saved in a directory irrelevant to the experiment version, and will be overwrite by next experiment. This is not acceptable.
### Pitch
I'm expecting the result hierarchy like this:
```
└── experiment_name
├── version_0
│ ├── config.yaml
│ └── checkpoints
│ └── epoch=7-step=3000.ckpt
└── version_1
├── config.yaml
└── checkpoints
└── epoch=28-step=10875.ckpt
```
I tried to fix this problem and was willingly to submit a PR. Fixing this issue seems more complicated than imagined.
### Additional context
Because I'm new to Lightning, I'm not sure I fully understand Logger's design. From the current code, several attributes such as `root_dir`, `log_dir`, and `save_dir` are not clearly defined. For example, in the defination of `lightning.fabric.loggers.tensorboard.TensorBoardLogger` class, the experiment name is not included in the `root_dir` path, but for `lightning.pytorch.loggers.tensorboard.TensorBoardLogger`, it is included.
The `root_dir` and `log_dir` properties can be `None` introduces unnecessary complexities. The documentation of `log_dir` says:
Return the root directory where all versions of an experiment get saved, or `None` if the logger does not save data locally.
If I understand it correctly, the property `root_dir` and `log_dir` should never be `None` for all concrete loggers. This is because the `config.yaml` should always be saved to somewhere like `root_dir/name/version`. Even if a logger does not save `config.yaml` locally, we can still initialize `root_dir` to the current directory ('.'), but not really use it.
I have two questions:
- Why we need the `save_dir` property? Why always setting `root_dir` to be `/name` and `log_dir` to be `/name/version` not sufficient?
- What makes `TensorBoardLogger` and `CSVLogger` special that result in the `Trainer.log_dir` to be `logger.log_dir` instead of `logger.save_dir`?
cc @lantiga @justusschock @borda @awaelchli
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the logger path properties in src/lightning/fabric/loggers/logger.py and src/lightning/pytorch/loggers/logger.py, then trace their use in src/lightning/pytorch/cli.py and callbacks/model_checkpoint.py. Review the TensorBoardLogger and CSVLogger behavior and existing path tests before deciding on consistent semantics. Done means each experiment version keeps config.yaml and checkpoints under one non-duplicated name/version hierarchy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- developer-experience
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100