Lightning-AI / Lightning-AI/pytorch-lightning
`TensorBoardLogger` fails with remote FS (azure)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
When using `TensorBoardLogger` with a `save_dir` pointing to azure blob storage, it will yield a `The blob type is invalid for this operation` error on log calls. Similar behaviour was reported in #18861 for `CSVLogger`.
### What version are you seeing the problem on?
v2.2
### How to reproduce the bug
```python
import lightning
from lightning.pytorch import loggers
from lightning.pytorch.demos import boring_classes
from typing import Any
log_dir = "az://@.blob.core.windows.net/test/"
class Model(boring_classes.BoringModel):
def training_step(self, batch: Any, batch_idx: int):
loss = self.step(batch)
self.log('loss', loss, on_step=True, logger=True)
self.logger.log_metrics({"loss": loss})
return {"loss": loss}
if __name__ == '__main__':
model = Model()
logger = loggers.TensorBoardLogger(log_dir, name="test")
trainer = lightning.Trainer(max_epochs=3, logger=logger)
trainer.fit(model)
```
### Error messages and logs
```
RequestId:6e66d9c5-d01e-0033-375a-8670de000000
Time:2024-04-04T06:33:45.2051891Z
ErrorCode:InvalidBlobType
Content: InvalidBlobTypeThe blob type is invalid for this operation.
RequestId:6e66d9c5-d01e-0033-375a-8670de000000
Time:2024-04-04T06:33:45.2051891Z
```
### Environment
Current environment
* CUDA:
- GPU: None
- available: False
- version: None
* Lightning:
- lightning: 2.2.1
- lightning-utilities: 0.10.1
- pytorch-lightning: 2.2.1
- torch: 2.2.1
- torchmetrics: 1.3.2
- torchvision: 0.17.1
* Packages:
- absl-py: 2.1.0
- adlfs: 2024.2.0
- aiohttp: 3.9.3
- aiosignal: 1.3.1
- antlr4-python3-runtime: 4.9.3
- argcomplete: 3.2.3
- async-timeout: 4.0.3
- attrs: 23.2.0
- azure-core: 1.30.1
- azure-datalake-store: 0.0.53
- azure-identity: 1.15.0
- azure-storage-blob: 12.19.1
- babel: 2.14.0
- bandit: 1.7.8
- black: 24.3.0
- certifi: 2024.2.2
- cffi: 1.16.0
- charset-normalizer: 3.3.2
- click: 8.1.7
- colorama: 0.4.6
- coloredlogs: 15.0.1
- colorlog: 6.8.2
- coverage: 7.4.4
- cryptography: 42.0.5
- distlib: 0.3.8
- exceptiongroup: 1.2.0
- filelock: 3.13.1
- flatbuffers: 24.3.7
- frozenlist: 1.4.1
- fsspec: 2024.3.0
- ghp-import: 2.1.0
- griffe: 0.42.0
- grpcio: 1.62.1
- h5py: 3.10.0
- huggingface-hub: 0.21.4
- humanfriendly: 10.0
- idna: 3.6
- importlib-metadata: 7.0.2
- importlib-resources: 6.3.2
- iniconfig: 2.0.0
- isodate: 0.6.1
- isort: 5.13.2
- jinja2: 3.1.3
- jsonargparse: 4.27.6
- kaiko-eva: 0.0.0.dev7+editable
- lightning: 2.2.1
- lightning-utilities: 0.10.1
- loguru: 0.7.2
- markdown: 3.5.2
- markdown-exec: 1.8.0
- markdown-it-py: 3.0.0
- markupsafe: 2.1.5
- mdurl: 0.1.2
- mergedeep: 1.3.4
- mike: 2.0.0
- mkdocs: 1.5.3
- mkdocs-autorefs: 1.0.1
- mkdocs-material: 9.5.14
- mkdocs-material-extensions: 1.3.1
- mkdocs-redirects: 1.2.1
- mkdocs-version-annotations: 1.0.0
- mkdocstrings: 0.24.1
- mkdocstrings-python: 1.9.0
- mpmath: 1.3.0
- msal: 1.28.0
- msal-extensions: 1.1.0
- multidict: 6.0.5
- mypy-extensions: 1.0.0
- networkx: 3.2.1
- nibabel: 5.2.1
- nodeenv: 1.8.0
- nox: 2024.3.2
- numpy: 1.26.4
- omegaconf: 2.3.0
- onnx: 1.15.0
- onnxruntime: 1.17.1
- opencv-python-headless: 4.9.0.80
- packaging: 24.0
- paginate: 0.5.6
- pandas: 2.2.1
- pathspec: 0.12.1
- pbr: 6.0.0
- pillow: 10.2.0
- pip: 24.0
- platformdirs: 4.2.0
- pluggy: 1.4.0
- portalocker: 2.8.2
- protobuf: 5.26.0
- pycparser: 2.21
- pygments: 2.17.2
- pyjwt: 2.8.0
- pymdown-extensions: 10.7.1
- pyparsing: 3.1.2
- pyright: 1.1.355
- pytest: 8.1.1
- pytest-cov: 4.1.0
- python-dateutil: 2.9.0.post0
- pytorch-lightning: 2.2.1
- pytz: 2024.1
- pyyaml: 6.0.1
- pyyaml-env-tag: 0.1
- regex: 2023.12.25
- requests: 2.31.0
- rich: 13.7.1
- ruff: 0.3.3
- safetensors: 0.4.2
- setuptools: 69.2.0
- six: 1.16.0
- stevedore: 5.2.0
- sympy: 1.12
- tensorboard: 2.16.2
- tensorboard-data-server: 0.7.2
- timm: 0.9.16
- tokenizers: 0.15.2
- tomli: 2.0.1
- toolz: 0.12.1
- torch: 2.2.1
- torchmetrics: 1.3.2
- torchvision: 0.17.1
- tqdm: 4.66.2
- transformers: 4.39.0
- typing-extensions: 4.10.0
- tzdata: 2024.1
- urllib3: 2.2.1
- verspec: 0.1.0
- virtualenv: 20.25.1
- watchdog: 4.0.0
- werkzeug: 3.0.1
- wheel: 0.42.0
- yamllint: 1.35.1
- yarl: 1.9.4
- zipp: 3.18.1
* System:
- OS: Darwin
- architecture:
- 64bit
-
- processor: arm
- python: 3.10.13
- release: 23.4.0
- version: Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000
### More info
_No response_
cc @lantiga
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the TensorBoardLogger entry point and reproduce the failure using the Azure blob save_dir shown in the issue. Trace the log calls that produce InvalidBlobType, then verify that TensorBoard metrics can be written successfully to Azure blob storage without the error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- cloud, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100