Lightning-AI / Lightning-AI/pytorch-lightning

`TensorBoardLogger` fails with remote FS (azure)

Open
#19,736 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug logger: tensorboard
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

When using `TensorBoardLogger` with a `save_dir` pointing to azure blob storage, it will yield a `The blob type is invalid for this operation` error on log calls. Similar behaviour was reported in #18861 for `CSVLogger`.

### What version are you seeing the problem on?

v2.2

### How to reproduce the bug

```python
import lightning
from lightning.pytorch import loggers
from lightning.pytorch.demos import boring_classes
from typing import Any

log_dir = "az://@.blob.core.windows.net/test/"

class Model(boring_classes.BoringModel):
def training_step(self, batch: Any, batch_idx: int):
loss = self.step(batch)
self.log('loss', loss, on_step=True, logger=True)
self.logger.log_metrics({"loss": loss})
return {"loss": loss}

if __name__ == '__main__':
model = Model()
logger = loggers.TensorBoardLogger(log_dir, name="test")
trainer = lightning.Trainer(max_epochs=3, logger=logger)
trainer.fit(model)
```

### Error messages and logs

```
RequestId:6e66d9c5-d01e-0033-375a-8670de000000
Time:2024-04-04T06:33:45.2051891Z
ErrorCode:InvalidBlobType
Content: InvalidBlobTypeThe blob type is invalid for this operation.
RequestId:6e66d9c5-d01e-0033-375a-8670de000000
Time:2024-04-04T06:33:45.2051891Z
```

### Environment

Current environment

* CUDA:
- GPU: None
- available: False
- version: None
* Lightning:
- lightning: 2.2.1
- lightning-utilities: 0.10.1
- pytorch-lightning: 2.2.1
- torch: 2.2.1
- torchmetrics: 1.3.2
- torchvision: 0.17.1
* Packages:
- absl-py: 2.1.0
- adlfs: 2024.2.0
- aiohttp: 3.9.3
- aiosignal: 1.3.1
- antlr4-python3-runtime: 4.9.3
- argcomplete: 3.2.3
- async-timeout: 4.0.3
- attrs: 23.2.0
- azure-core: 1.30.1
- azure-datalake-store: 0.0.53
- azure-identity: 1.15.0
- azure-storage-blob: 12.19.1
- babel: 2.14.0
- bandit: 1.7.8
- black: 24.3.0
- certifi: 2024.2.2
- cffi: 1.16.0
- charset-normalizer: 3.3.2
- click: 8.1.7
- colorama: 0.4.6
- coloredlogs: 15.0.1
- colorlog: 6.8.2
- coverage: 7.4.4
- cryptography: 42.0.5
- distlib: 0.3.8
- exceptiongroup: 1.2.0
- filelock: 3.13.1
- flatbuffers: 24.3.7
- frozenlist: 1.4.1
- fsspec: 2024.3.0
- ghp-import: 2.1.0
- griffe: 0.42.0
- grpcio: 1.62.1
- h5py: 3.10.0
- huggingface-hub: 0.21.4
- humanfriendly: 10.0
- idna: 3.6
- importlib-metadata: 7.0.2
- importlib-resources: 6.3.2
- iniconfig: 2.0.0
- isodate: 0.6.1
- isort: 5.13.2
- jinja2: 3.1.3
- jsonargparse: 4.27.6
- kaiko-eva: 0.0.0.dev7+editable
- lightning: 2.2.1
- lightning-utilities: 0.10.1
- loguru: 0.7.2
- markdown: 3.5.2
- markdown-exec: 1.8.0
- markdown-it-py: 3.0.0
- markupsafe: 2.1.5
- mdurl: 0.1.2
- mergedeep: 1.3.4
- mike: 2.0.0
- mkdocs: 1.5.3
- mkdocs-autorefs: 1.0.1
- mkdocs-material: 9.5.14
- mkdocs-material-extensions: 1.3.1
- mkdocs-redirects: 1.2.1
- mkdocs-version-annotations: 1.0.0
- mkdocstrings: 0.24.1
- mkdocstrings-python: 1.9.0
- mpmath: 1.3.0
- msal: 1.28.0
- msal-extensions: 1.1.0
- multidict: 6.0.5
- mypy-extensions: 1.0.0
- networkx: 3.2.1
- nibabel: 5.2.1
- nodeenv: 1.8.0
- nox: 2024.3.2
- numpy: 1.26.4
- omegaconf: 2.3.0
- onnx: 1.15.0
- onnxruntime: 1.17.1
- opencv-python-headless: 4.9.0.80
- packaging: 24.0
- paginate: 0.5.6
- pandas: 2.2.1
- pathspec: 0.12.1
- pbr: 6.0.0
- pillow: 10.2.0
- pip: 24.0
- platformdirs: 4.2.0
- pluggy: 1.4.0
- portalocker: 2.8.2
- protobuf: 5.26.0
- pycparser: 2.21
- pygments: 2.17.2
- pyjwt: 2.8.0
- pymdown-extensions: 10.7.1
- pyparsing: 3.1.2
- pyright: 1.1.355
- pytest: 8.1.1
- pytest-cov: 4.1.0
- python-dateutil: 2.9.0.post0
- pytorch-lightning: 2.2.1
- pytz: 2024.1
- pyyaml: 6.0.1
- pyyaml-env-tag: 0.1
- regex: 2023.12.25
- requests: 2.31.0
- rich: 13.7.1
- ruff: 0.3.3
- safetensors: 0.4.2
- setuptools: 69.2.0
- six: 1.16.0
- stevedore: 5.2.0
- sympy: 1.12
- tensorboard: 2.16.2
- tensorboard-data-server: 0.7.2
- timm: 0.9.16
- tokenizers: 0.15.2
- tomli: 2.0.1
- toolz: 0.12.1
- torch: 2.2.1
- torchmetrics: 1.3.2
- torchvision: 0.17.1
- tqdm: 4.66.2
- transformers: 4.39.0
- typing-extensions: 4.10.0
- tzdata: 2024.1
- urllib3: 2.2.1
- verspec: 0.1.0
- virtualenv: 20.25.1
- watchdog: 4.0.0
- werkzeug: 3.0.1
- wheel: 0.42.0
- yamllint: 1.35.1
- yarl: 1.9.4
- zipp: 3.18.1
* System:
- OS: Darwin
- architecture:
- 64bit
-
- processor: arm
- python: 3.10.13
- release: 23.4.0
- version: Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000

### More info

_No response_

cc @lantiga

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the TensorBoardLogger entry point and reproduce the failure using the Azure blob save_dir shown in the issue. Trace the log calls that produce InvalidBlobType, then verify that TensorBoard metrics can be written successfully to Azure blob storage without the error.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
cloud, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.