Lightning-AI / Lightning-AI/pytorch-lightning
Syncing the log_dir across ranks is not valid with multiple nodes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## 🐛 Bug
In TrainerProperties, we [compute the log_dir, and then we overwrite it with rank 0's log_dir](https://github.com/PyTorchLightning/pytorch-lightning/blob/8c0ea92af237542f5b36ae684543f871da829379/pytorch_lightning/trainer/properties.py#L137) when returning the log_dir. This isn't valid in a multi-node setup because rank 0's log_dir may not exist on rank n's machine.
### To Reproduce
We can use any model for this bug.
With 2 nodes
1. Create a unique temporary directory on each node as log_dir
2. Pass the unique log_dir into one of the lightning loggers and pass it into the trainer on each node
3. (possibly optional) pass a profiler into the trainer (to request log_dir and try to write to it)
3. Observe failure when we attempt to write something the log_dir on node 2
### Expected behavior
If I pass a log_dir into the Logger or default_root_dir into the trainer, it should use that value.
### Environment
- PyTorch Lightning Version (e.g., 1.3.0): 1.3.8 (still exists on master)
- PyTorch Version (e.g., 1.8) 1.6
- Python version: 3.6.9
- OS (e.g., Linux): Linux
- CUDA/cuDNN version: 10.1
- GPU models and configuration: Any model. 1 GPU per node, 2 nodes
- How you installed PyTorch (`conda`, `pip`, source): conda
- If compiling from source, the output of `torch.__config__.show()`:
- Any other relevant information:
### Additional context
https://pytorch-lightning.slack.com/archives/CRBLFHY79/p1630348135165800
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.