Lightning-AI / Lightning-AI/pytorch-lightning

MLFlowLogger.save_dir mishandles absolute file: URIs on Windows

Open
#20,972 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug logger logger: mlflow ver: 2.5.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

When passing an absolute file path as tracking_uri on Windows, MLFlowLogger.save_dir does not handle the file: URI correctly, causing a malformed local path.

This results in writing to an invalid path like ///C:/..., resulting in FileNotFoundError: [WinError 161] The specified path is invalid.

I am aware that using mlflow.set_tracking_uri("http://localhost:8080") or mlflow.set_tracking_uri("file:./mlruns") is a valid workaround.

This is not related to #20669. I am willing to contribute a fix and a test case if needed.

```python
# example

import mlflow
from lightning.pytorch import Trainer
from lightning.pytorch.demos import BoringModel
from lightning.pytorch.loggers import MLFlowLogger

model = BoringModel()

mlflow.pytorch.autolog()
with mlflow.start_run() as run:
logger = MLFlowLogger(
tracking_uri=mlflow.get_tracking_uri(), # file:///C:/Dev/example/mlruns
run_id=run.info.run_id,
)

trainer = Trainer(
max_epochs=1,
logger=logger,
limit_train_batches=1,
limit_val_batches=1,
)

trainer.fit(model)
```

```python
# this code mishandles absolute paths on Windows
# tracking_uri
# file:///C:/Dev/example/mlruns
# result:
# ///C:/Dev/example/mlruns
# expected:
# C:/Dev/example/mlruns

@property
@override
def save_dir(self) -> Optional[str]:
"""The root file directory in which MLflow experiments are saved.

Return:
Local path to the root experiment directory if the tracking uri is local.
Otherwise returns `None`.

"""
if self._tracking_uri.startswith(LOCAL_FILE_URI_PREFIX):
return self._tracking_uri[len(LOCAL_FILE_URI_PREFIX) :]
return None
```

```python
# suggested fix

@property
@override
def save_dir(self) -> Optional[str]:
"""The root file directory in which MLflow experiments are saved.

Return:
Local path to the root experiment directory if the tracking uri is local.
Otherwise returns `None`.

"""
from urllib.parse import urlparse
from urllib.request import url2pathname

if self._tracking_uri.startswith(LOCAL_FILE_URI_PREFIX):
p = urlparse(self._tracking_uri)
return url2pathname(p.path)
return None
```

### What version are you seeing the problem on?

v2.5

### Reproduced in studio

_No response_

### How to reproduce the bug

```python

```

### Error messages and logs

```
Traceback (most recent call last):
File "C:\Dev\example\main.py", line 23, in
trainer.fit(model)
File "C:\Dev\example\venv\Lib\site-packages\mlflow\utils\autologging_utils\safety.py", line 484, in safe_patch_function
patch_function(call_original, *args, **kwargs)
File "C:\Dev\example\venv\Lib\site-packages\mlflow\utils\autologging_utils\safety.py", line 182, in patch_with_managed_run
result = patch_function(original, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Dev\example\venv\Lib\site-packages\mlflow\pytorch\_lightning_autolog.py", line 544, in patched_fit
result = original(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Dev\example\venv\Lib\site-packages\mlflow\utils\autologging_utils\safety.py", line 475, in call_original
return call_original_fn_with_event_logging(_original_fn, og_args, og_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Dev\example\venv\Lib\site-packages\mlflow\utils\autologging_utils\safety.py", line 426, in call_original_fn_with_event_logging
original_fn_result = original_fn(*og_args, **og_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Dev\example\venv\Lib\site-packages\mlflow\utils\autologging_utils\safety.py", line 472, in _original_fn
original_result = original(*_og_args, **_og_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Dev\example\venv\Lib\site-packages\lightning\pytorch\trainer\trainer.py", line 561, in fit
call._call_and_handle_interrupt(
File "C:\Dev\example\venv\Lib\site-packages\lightning\pytorch\trainer\call.py", line 48, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Dev\example\venv\Lib\site-packages\lightning\pytorch\trainer\trainer.py", line 599, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "C:\Dev\example\venv\Lib\site-packages\lightning\pytorch\trainer\trainer.py", line 1012, in _run
results = self._run_stage()
^^^^^^^^^^^^^^^^^
File "C:\Dev\example\venv\Lib\site-packages\lightning\pytorch\trainer\trainer.py", line 1056, in _run_stage
self.fit_loop.run()
File "C:\Dev\example\venv\Lib\site-packages\lightning\pytorch\loops\fit_loop.py", line 217, in run
self.on_advance_end()
File "C:\Dev\example\venv\Lib\site-packages\lightning\pytorch\loops\fit_loop.py", line 470, in on_advance_end
call._call_callback_hooks(trainer, "on_train_epoch_end", monitoring_callbacks=True)
File "C:\Dev\example\venv\Lib\site-packages\lightning\pytorch\trainer\call.py", line 227, in _call_callback_hooks
fn(trainer, trainer.lightning_module, *args, **kwargs)
File "C:\Dev\example\venv\Lib\site-packages\lightning\pytorch\callbacks\model_checkpoint.py", line 329, in on_train_epoch_end
self._save_topk_checkpoint(trainer, monitor_candidates)
File "C:\Dev\example\venv\Lib\site-packages\lightning\pytorch\callbacks\model_checkpoint.py", line 391, in _save_topk_checkpoint
self._save_none_monitor_checkpoint(trainer, monitor_candidates)
File "C:\Dev\example\venv\Lib\site-packages\lightning\pytorch\callbacks\model_checkpoint.py", line 719, in _save_none_monitor_checkpoint
self._save_checkpoint(trainer, filepath)
File "C:\Dev\example\venv\Lib\site-packages\lightning\pytorch\callbacks\model_checkpoint.py", line 394, in _save_checkpoint
trainer.save_checkpoint(filepath, self.save_weights_only)
File "C:\Dev\example\venv\Lib\site-packages\lightning\pytorch\trainer\trainer.py", line 1397, in save_checkpoint
self.strategy.save_checkpoint(checkpoint, filepath, storage_options=storage_options)
File "C:\Dev\example\venv\Lib\site-packages\lightning\pytorch\strategies\strategy.py", line 491, in save_checkpoint
self.checkpoint_io.save_checkpoint(checkpoint, filepath, storage_options=storage_options)
File "C:\Dev\example\venv\Lib\site-packages\lightning\fabric\plugins\io\torch_io.py", line 57, in save_checkpoint
fs.makedirs(os.path.dirname(path), exist_ok=True)
File "C:\Dev\example\venv\Lib\site-packages\fsspec\implementations\local.py", line 53, in makedirs
os.makedirs(path, exist_ok=exist_ok)
File "", line 215, in makedirs
File "", line 215, in makedirs
File "", line 215, in makedirs
[Previous line repeated 3 more times]
File "", line 225, in makedirs
FileNotFoundError: [WinError 161] The specified path is invalid: '///C:/'
```

### Environment

Current environment

```
#- PyTorch Lightning Version (e.g., 2.5.0): 2.5.2
#- PyTorch Version (e.g., 2.5): 2.7.1
#- Python version (e.g., 3.12): 3.12.4
#- OS (e.g., Linux): Windows 10
#- CUDA/cuDNN version: N/A
#- GPU models and configuration: N/A
#- How you installed Lightning(`conda`, `pip`, source): pip
```

### More info

_No response_

cc @lantiga @borda

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at MLFlowLogger.save_dir, using the property implementation and Windows example in the issue to trace how the file: URI is converted into a local path. Reproduce the case with file:///C:/Dev/example/mlruns and verify that the resulting save directory is C:/Dev/example/mlruns rather than ///C:/....

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.