Lightning-AI / Lightning-AI/pytorch-lightning

Syncing the log_dir across ranks is not valid with multiple nodes

Open
#9,212 3 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

bug distributed help wanted logging priority: 1
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

## 🐛 Bug

In TrainerProperties, we [compute the log_dir, and then we overwrite it with rank 0's log_dir](https://github.com/PyTorchLightning/pytorch-lightning/blob/8c0ea92af237542f5b36ae684543f871da829379/pytorch_lightning/trainer/properties.py#L137) when returning the log_dir. This isn't valid in a multi-node setup because rank 0's log_dir may not exist on rank n's machine.

### To Reproduce

We can use any model for this bug.

With 2 nodes
1. Create a unique temporary directory on each node as log_dir
2. Pass the unique log_dir into one of the lightning loggers and pass it into the trainer on each node
3. (possibly optional) pass a profiler into the trainer (to request log_dir and try to write to it)
3. Observe failure when we attempt to write something the log_dir on node 2

### Expected behavior

If I pass a log_dir into the Logger or default_root_dir into the trainer, it should use that value.

### Environment

- PyTorch Lightning Version (e.g., 1.3.0): 1.3.8 (still exists on master)
- PyTorch Version (e.g., 1.8) 1.6
- Python version: 3.6.9
- OS (e.g., Linux): Linux
- CUDA/cuDNN version: 10.1
- GPU models and configuration: Any model. 1 GPU per node, 2 nodes
- How you installed PyTorch (`conda`, `pip`, source): conda
- If compiling from source, the output of `torch.__config__.show()`:
- Any other relevant information:

### Additional context

https://pytorch-lightning.slack.com/archives/CRBLFHY79/p1630348135165800

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.