Lightning-AI / Lightning-AI/pytorch-lightning

TensorboardLogger not handling remote filesystem paths correctly on Windows environment

Open
#17,601 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug help wanted logger logger: tensorboard ver: 2.0.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

When attempting to use a remote filesystem, the TensorboardLogger does not handle the name and version parameters correctly. The issue seems to occur specifically when running training scripts from Windows environments. The problem arises because the os.path.join() function tries to concatenate the different URI parts using Windows backslash characters (\) instead of forward slashes (/).

One possible solution to avoid this issue is to first check the filesystem being used (while parsing the save_dir logger parameter) and then employ the appropriate path concatenation strategy if a remote filesystem is detected, regardless of the operating system executing the script.

### What version are you seeing the problem on?

v2.0

### How to reproduce the bug

Here's an example using S3 (in my case, self-hosted in a Minio Docker container):

1. Run the following code from a **Windows environment** (ensure that **my_bucket** is already created, as failure to format the URI correctly may result in an **InvalidBucketName** issue during bucket creation):

```python
logger = TensorBoardLogger(save_dir='s3://my_bucket/logs')
```
2. Notice that in your 'my_bucket' bucket, there is a single folder named 'logs\lightning_logs\version_0' instead of a sequence of nested folders ('logs', 'lightning_logs', and 'version_0').

### Error messages and logs

No error messages or logs are generated. In my Minio bucket, paths are not split into multiple directories as expected when using TensorboardLogger with S3. Although I haven't checked other filesystems, I believe they may suffer from the same issue. Instead of creating multiple directories, a single directory is created with a long name that includes all the subdirectories.

### Environment

Current environment

```
#- Lightning Componen: Trainer, TensorboardLogger:
#- PyTorch Lightning Version: 2.0.2
#- PyTorch Version: 1.13.1
#- Python version: 3.10.9
#- OS: Windows
#- How you installed Lightning : poetry
```

### More info

_No response_

cc @awaelchli @borda @Blaizzy

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with TensorboardLogger's save_dir parsing and reproduce the issue on Windows using the provided S3 example. Confirm that the remote path is split into nested logs, lightning_logs, and version_0 directories with forward slashes rather than one backslash-containing directory.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.