Lightning-AI / Lightning-AI/pytorch-lightning
Incompatibility with MLFlow 1.30
@awaelchli is already working on this.
Since Mar 15, 2023.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
This is just to let you guys know that there's an incompatibility bug with MLFlow 1.30.
PL creates runs through [this line](https://github.com/Lightning-AI/lightning/blob/caa3329b79965f7712e49bd9f8933f7a4686c49b/src/pytorch_lightning/loggers/mlflow.py#L195) without specifying the run_name, which will make MLFlow generate a random run_name [here](https://github.com/mlflow/mlflow/blob/d975b5a19c8b04db97ba446cce91eb876f8ee60e/mlflow/store/tracking/file_store.py#L607) and then this will be added to the tags [here](https://github.com/mlflow/mlflow/blob/d975b5a19c8b04db97ba446cce91eb876f8ee60e/mlflow/store/tracking/file_store.py#L632). But if a run_name has already been passed through PL with the tags, this will create a duplication error.
This is fixed in MLFlow 2.0. So could you perhaps change the version requirement, or add run_name to the create_run line in src/pytorch_lightning/loggers/mlflow.py? Or if someone can investigate further it would be great too. Thanks!
### How to reproduce the bug
```python
- start an experiment with run_name specified in the config, and log with MLFlow.
```
### Error messages and logs
```
Traceback (most recent call last):
File "run_finetune.py", line 104, in main
logger.log_hyperparams({'trainer': OmegaConf.to_object(cfg.trainer)})
File "***/lib/python3.8/site-packages/lightning_utilities/core/rank_zero.py", line 27, in wrapped_fn
return fn(*args, **kwargs)
File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/mlflow.py", line 231, in log_hyperparams
self.experiment.log_param(self.run_id, k, v)
File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/logger.py", line 53, in experiment
return get_experiment() or DummyExperiment()
File "***/lib/python3.8/site-packages/lightning_utilities/core/rank_zero.py", line 27, in wrapped_fn
return fn(*args, **kwargs)
File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/logger.py", line 51, in get_experiment
return fn(self)
File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/mlflow.py", line 195, in experiment
run = self._mlflow_client.create_run(experiment_id=self._experiment_id, tags=resolve_tags(self.tags))
File "***/lib/python3.8/site-packages/mlflow/tracking/client.py", line 270, in create_run
return self._tracking_client.create_run(experiment_id, start_time, tags, run_name)
File "***/lib/python3.8/site-packages/mlflow/tracking/_tracking_service/client.py", line 108, in create_run
return self.store.create_run(
File "***/lib/python3.8/site-packages/mlflow/store/tracking/rest_store.py", line 204, in create_run
response_proto = self._call_endpoint(CreateRun, req_body)
File "***/lib/python3.8/site-packages/mlflow/store/tracking/rest_store.py", line 57, in _call_endpoint
return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
File "***/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 280, in call_endpoint
response = verify_rest_response(response, endpoint)
File "***/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 206, in verify_rest_response
raise RestException(json.loads(response.text))
mlflow.exceptions.RestException: BAD_REQUEST: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "tag_pk"
DETAIL: Key (key, run_uuid)=(mlflow.runName, 8****) already exists.
[SQL: INSERT INTO tags (key, value, run_uuid) VALUES (%(key)s, %(value)s, %(run_uuid)s)]
[parameters: (****{'key': 'mlflow.runName', 'value': 'finetune_trial', 'run_uuid': '8****'}, {'key': 'mlflow.runName', 'value': 'bouncy-squid-39', 'run_uuid': '8****'})]
```
### Environment
Current environment
```
PyTorch Lightning 1.8.6
MLFlow 1.30.0
```
### More info
_No response_
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.