Lightning-AI / Lightning-AI/pytorch-lightning
MLFlow logger with remote tracking fails with CLI
Open
Nobody has claimed this yet.
bug
help wanted
lightningcli
logger: mlflow
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
Bug description
Running with the LightningCLI, MLflow logger, and MLFLOW_TRACKING_URI environment variable set causes an assertion failure with logging. I think using a remote tracking server causes no local log files to be created which the CLI doesn't like.
I suspect it's a similar issue to #12748.
How to reproduce the bug
from pytorch_lightning.cli import LightningCLI
from helpers import BoringModel, BoringDataModule
cli = LightningCLI(
BoringModel,
BoringDataModule,
trainer_defaults=dict(
max_epochs=1,
logger="pytorch_lightning.loggers.MLFlowLogger"
)
)
$ mlflow server
...
$ MLFLOW_TRACKING_URI=http://localhost:5000 python main.py fit
Error messages and logs
/home/vscode/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/configuration_validator.py:106: UserWarning: You passed in a `val_dataloader` but have no `validation_step`. Skipping val loop.
rank_zero_warn("You passed in a `val_dataloader` but have no `validation_step`. Skipping val loop.")
Traceback (most recent call last):
File "/workspaces/mlflow_log_error/main.py", line 4, in <module>
cli = LightningCLI(
File "/home/vscode/.local/lib/python3.10/site-packages/pytorch_lightning/cli.py", line 354, in __init__
self._run_subcommand(self.subcommand)
File "/home/vscode/.local/lib/python3.10/site-packages/pytorch_lightning/cli.py", line 665, in _run_subcommand
fn(**fn_kwargs)
File "/home/vscode/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit
call._call_and_handle_interrupt(
File "/home/vscode/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/vscode/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 645, in _fit_impl
self._run(model, ckpt_path=self.ckpt_path)
File "/home/vscode/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1037, in _run
self._call_setup_hook() # allow user to setup lightning_module in accelerator environment
File "/home/vscode/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1285, in _call_setup_hook
self._call_callback_hooks("setup", stage=fn)
File "/home/vscode/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1380, in _call_callback_hooks
fn(self, self.lightning_module, *args, **kwargs)
File "/home/vscode/.local/lib/python3.10/site-packages/pytorch_lightning/cli.py", line 216, in setup
assert log_dir is not None
AssertionError
Environment
* CUDA:
- GPU: None
- available: False
- version: 11.7
* Lightning:
- lightning: 1.9.0rc0
- lightning-cloud: 0.5.16
- lightning-utilities: 0.5.0
- pytorch-lightning: 1.8.6
- torch: 1.13.1
- torchmetrics: 0.11.0
- torchvision: 0.14.1
* Packages:
- aiohttp: 3.8.3
- aiosignal: 1.3.1
- alembic: 1.9.1
- antlr4-python3-runtime: 4.9.3
- anyio: 3.6.2
- argon2-cffi: 21.3.0
- argon2-cffi-bindings: 21.2.0
- arrow: 1.2.3
- asttokens: 2.2.1
- async-timeout: 4.0.2
- attrs: 22.2.0
- babel: 2.11.0
- backcall: 0.2.0
- beautifulsoup4: 4.11.1
- bleach: 5.0.1
- blessed: 1.19.1
- build: 0.9.0
- certifi: 2022.12.7
- cffi: 1.15.1
- charset-normalizer: 2.1.1
- click: 8.1.3
- cloudpickle: 2.2.0
- comm: 0.1.2
- commonmark: 0.9.1
- contourpy: 1.0.6
- croniter: 1.3.8
- cycler: 0.11.0
- databricks-cli: 0.17.4
- dateutils: 0.6.12
- debugpy: 1.6.5
- decorator: 5.1.1
- deepdiff: 6.2.3
- defusedxml: 0.7.1
- dnspython: 2.2.1
- docker: 6.0.1
- docstring-parser: 0.15
- email-validator: 1.3.0
- entrypoints: 0.4
- executing: 1.2.0
- fastapi: 0.88.0
- fastjsonschema: 2.16.2
- flask: 2.2.2
- fonttools: 4.38.0
- fqdn: 1.5.1
- frozenlist: 1.3.3
- fsspec: 2022.11.0
- gitdb: 4.0.10
- gitpython: 3.1.30
- greenlet: 2.0.1
- gunicorn: 20.1.0
- h11: 0.14.0
- httpcore: 0.16.3
- httptools: 0.5.0
- httpx: 0.23.3
- hydra-core: 1.3.1
- idna: 3.4
- importlib-metadata: 5.2.0
- importlib-resources: 5.10.2
- inquirer: 3.1.2
- ipykernel: 6.20.1
- ipython: 8.8.0
- ipython-genutils: 0.2.0
- isoduration: 20.11.0
- itsdangerous: 2.1.2
- jedi: 0.18.2
- jinja2: 3.1.2
- joblib: 1.2.0
- json5: 0.9.11
- jsonargparse: 4.19.0
- jsonpointer: 2.3
- jsonschema: 4.17.3
- jupyter-client: 7.4.8
- jupyter-core: 5.1.3
- jupyter-events: 0.6.0
- jupyter-server: 2.0.6
- jupyter-server-terminals: 0.4.4
- jupyterlab: 3.5.2
- jupyterlab-pygments: 0.2.2
- jupyterlab-server: 2.18.0
- kiwisolver: 1.4.4
- lightning: 1.9.0rc0
- lightning-cloud: 0.5.16
- lightning-utilities: 0.5.0
- llvmlite: 0.39.1
- mako: 1.2.4
- markdown: 3.4.1
- markupsafe: 2.1.1
- matplotlib: 3.6.2
- matplotlib-inline: 0.1.6
- mistune: 2.0.4
- mlflow: 2.1.1
- multidict: 6.0.4
- nbclassic: 0.4.8
- nbclient: 0.7.2
- nbconvert: 7.2.7
- nbformat: 5.7.1
- nest-asyncio: 1.5.6
- notebook: 6.5.2
- notebook-shim: 0.2.2
- numba: 0.56.4
- numpy: 1.23.5
- nvidia-cublas-cu11: 11.10.3.66
- nvidia-cuda-nvrtc-cu11: 11.7.99
- nvidia-cuda-runtime-cu11: 11.7.99
- nvidia-cudnn-cu11: 8.5.0.96
- oauthlib: 3.2.2
- omegaconf: 2.3.0
- ordered-set: 4.1.0
- orjson: 3.8.4
- packaging: 21.3
- pandas: 1.5.2
- pandocfilters: 1.5.0
- parso: 0.8.3
- pep517: 0.13.0
- pexpect: 4.8.0
- pickleshare: 0.7.5
- pillow: 9.4.0
- pip: 22.3.1
- pip-tools: 6.12.1
- platformdirs: 2.6.2
- prometheus-client: 0.15.0
- prompt-toolkit: 3.0.36
- protobuf: 3.20.1
- psutil: 5.9.4
- ptyprocess: 0.7.0
- pure-eval: 0.2.2
- pyarrow: 10.0.1
- pycparser: 2.21
- pydantic: 1.10.4
- pygments: 2.14.0
- pyjwt: 2.6.0
- pyparsing: 3.0.9
- pyrsistent: 0.19.3
- python-dateutil: 2.8.2
- python-dotenv: 0.21.0
- python-editor: 1.0.4
- python-json-logger: 2.0.4
- python-multipart: 0.0.5
- pytorch-lightning: 1.8.6
- pytz: 2022.7
- pyyaml: 6.0
- pyzmq: 24.0.1
- querystring-parser: 1.2.4
- readchar: 4.0.3
- requests: 2.28.1
- rfc3339-validator: 0.1.4
- rfc3986: 1.5.0
- rfc3986-validator: 0.1.1
- rich: 13.0.1
- scikit-learn: 1.2.0
- scipy: 1.10.0
- send2trash: 1.8.0
- setuptools: 65.5.0
- shap: 0.41.0
- six: 1.16.0
- slicer: 0.0.7
- smmap: 5.0.0
- sniffio: 1.3.0
- soupsieve: 2.3.2.post1
- sqlalchemy: 1.4.46
- sqlparse: 0.4.3
- stack-data: 0.6.2
- starlette: 0.22.0
- starsessions: 1.3.0
- tabulate: 0.9.0
- tensorboardx: 2.5.1
- terminado: 0.17.1
- threadpoolctl: 3.1.0
- tinycss2: 1.2.1
- tomli: 2.0.1
- torch: 1.13.1
- torchmetrics: 0.11.0
- torchvision: 0.14.1
- tornado: 6.2
- tqdm: 4.64.1
- traitlets: 5.8.1
- typeshed-client: 2.1.0
- typing-extensions: 4.4.0
- ujson: 5.7.0
- uri-template: 1.2.0
- urllib3: 1.26.13
- uvicorn: 0.20.0
- uvloop: 0.17.0
- watchfiles: 0.18.1
- wcwidth: 0.2.5
- webcolors: 1.12
- webencodings: 0.5.1
- websocket-client: 1.4.2
- websockets: 10.4
- werkzeug: 2.2.2
- wheel: 0.38.4
- yarl: 1.8.2
- zipp: 3.11.0
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor:
- python: 3.10.9
- version: #1 SMP Wed Mar 2 00:30:59 UTC 2022
More info
No response
cc @carmocca @mauvilsa
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at pytorch_lightning/cli.py, especially LightningCLI.setup where the assertion at line 216 is raised, and compare the remote MLFlowLogger path with issue #12748. Reproduce the provided LightningCLI command with MLFLOW_TRACKING_URI set; done means the CLI completes without the AssertionError when using a remote tracking server.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100