Lightning-AI / Lightning-AI/pytorch-lightning

Incompatibility with MLFlow 1.30

Open
#17,084 4 comments 0 reactions 1 assignee View on GitHub

@awaelchli is already working on this.

Since Mar 15, 2023.

3rd party bug help wanted logger: mlflow
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

This is just to let you guys know that there's an incompatibility bug with MLFlow 1.30.

PL creates runs through [this line](https://github.com/Lightning-AI/lightning/blob/caa3329b79965f7712e49bd9f8933f7a4686c49b/src/pytorch_lightning/loggers/mlflow.py#L195) without specifying the run_name, which will make MLFlow generate a random run_name [here](https://github.com/mlflow/mlflow/blob/d975b5a19c8b04db97ba446cce91eb876f8ee60e/mlflow/store/tracking/file_store.py#L607) and then this will be added to the tags [here](https://github.com/mlflow/mlflow/blob/d975b5a19c8b04db97ba446cce91eb876f8ee60e/mlflow/store/tracking/file_store.py#L632). But if a run_name has already been passed through PL with the tags, this will create a duplication error.

This is fixed in MLFlow 2.0. So could you perhaps change the version requirement, or add run_name to the create_run line in src/pytorch_lightning/loggers/mlflow.py? Or if someone can investigate further it would be great too. Thanks!

### How to reproduce the bug

```python
- start an experiment with run_name specified in the config, and log with MLFlow.
```

### Error messages and logs

```
Traceback (most recent call last):
File "run_finetune.py", line 104, in main
logger.log_hyperparams({'trainer': OmegaConf.to_object(cfg.trainer)})
File "***/lib/python3.8/site-packages/lightning_utilities/core/rank_zero.py", line 27, in wrapped_fn
return fn(*args, **kwargs)
File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/mlflow.py", line 231, in log_hyperparams
self.experiment.log_param(self.run_id, k, v)
File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/logger.py", line 53, in experiment
return get_experiment() or DummyExperiment()
File "***/lib/python3.8/site-packages/lightning_utilities/core/rank_zero.py", line 27, in wrapped_fn
return fn(*args, **kwargs)
File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/logger.py", line 51, in get_experiment
return fn(self)
File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/mlflow.py", line 195, in experiment
run = self._mlflow_client.create_run(experiment_id=self._experiment_id, tags=resolve_tags(self.tags))
File "***/lib/python3.8/site-packages/mlflow/tracking/client.py", line 270, in create_run
return self._tracking_client.create_run(experiment_id, start_time, tags, run_name)
File "***/lib/python3.8/site-packages/mlflow/tracking/_tracking_service/client.py", line 108, in create_run
return self.store.create_run(
File "***/lib/python3.8/site-packages/mlflow/store/tracking/rest_store.py", line 204, in create_run
response_proto = self._call_endpoint(CreateRun, req_body)
File "***/lib/python3.8/site-packages/mlflow/store/tracking/rest_store.py", line 57, in _call_endpoint
return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
File "***/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 280, in call_endpoint
response = verify_rest_response(response, endpoint)
File "***/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 206, in verify_rest_response
raise RestException(json.loads(response.text))
mlflow.exceptions.RestException: BAD_REQUEST: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "tag_pk"
DETAIL: Key (key, run_uuid)=(mlflow.runName, 8****) already exists.

[SQL: INSERT INTO tags (key, value, run_uuid) VALUES (%(key)s, %(value)s, %(run_uuid)s)]
[parameters: (****{'key': 'mlflow.runName', 'value': 'finetune_trial', 'run_uuid': '8****'}, {'key': 'mlflow.runName', 'value': 'bouncy-squid-39', 'run_uuid': '8****'})]

```

### Environment

Current environment

```
PyTorch Lightning 1.8.6
MLFlow 1.30.0
```

### More info

_No response_

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.