Lightning-AI / Lightning-AI/pytorch-lightning

Mlflow logging LR duplicate key issue with PostgreSQL DB #190

Open
#20,865 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug logger: mlflow ver: 2.5.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

**What happened?**
In the anemoi codebase - we use the pytorch lightning MlflowLogger and we hit this error when logging this to a remote server with a postgresql database:

`mlflow.exceptions.RestException: BAD_REQUEST: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely)
(psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "metric_pk"
DETAIL: Key (key, "timestamp", step, run_uuid, value, is_nan)=(lr-AdamW, 1741862947817, 60899, 843c1331fefa436bab56485cfc5bc16e, 0.000322870392593084, f) already exists.`

https://github.com/ecmwf/anemoi-core/blob/main/training/src/anemoi/training/diagnostics/mlflow/logger.py

versions:

```
mlflow 2.22.0
pytorch-lightning 2.5.1
```

### What version are you seeing the problem on?

v2.5

### Reproduced in studio

_No response_

### How to reproduce the bug

```python

```

### Error messages and logs

```
# Error messages and logs here please
```

### Environment

Current environment

```
#- PyTorch Lightning Version (e.g., 2.5.0):
#- PyTorch Version (e.g., 2.5):
#- Python version (e.g., 3.12):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
```

### More info

_No response_

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with training/src/anemoi/training/diagnostics/mlflow/logger.py and review how the MlflowLogger records the lr-AdamW metric. Reproduce the reported duplicate-key error against a remote PostgreSQL database using MLflow 2.22.0 and PyTorch Lightning 2.5.1; done means the metric can be logged without the metric_pk violation.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python, pytorch
Domain
databases, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.