Lightning-AI / Lightning-AI/pytorch-lightning

[CLI] predict(ckpt_path="best") raises ValueError even though ModelCheckpoint is configured under trainer.callbacks

Open
#21,254 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug checkpointing lightningcli trainer: predict ver: 2.5.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

When running trainer.predict via LightningCLI with --ckpt_path best, Lightning raises:
```bash
ValueError: `.predict(ckpt_path="best")` is set but `ModelCheckpoint` is not configured to save the best model.
```

However, ModelCheckpoint is configured under trainer.callbacks in the YAML, and training is executed via LightningCLI. This looks like either:
1. LightningCLI isn’t detecting the ModelCheckpoint from the CLI YAML when resolving best during predict, or
2. The checkpoint_connector’s detection logic for best doesn’t handle the CLI/YAML path or log directory resolution, or
3. A mismatch between monitor metric registration and best-checkpoint discovery is not surfaced clearly (i.e., the callback exists, but the condition for “best” was silently unmet).

### What version are you seeing the problem on?

v2.5

### Reproduced in studio

_No response_

### How to reproduce the bug

```python
trainer:
callbacks:
- class_path: lightning.pytorch.callbacks.ModelCheckpoint
init_args:
monitor: val_loss
mode: min
save_top_k: 1
filename: "{val_loss:.4f}"
save_last: false

uv run src/cli.py predict -c config.yaml --ckpt_path=best
```

### Error messages and logs

```
# Error messages and logs here please
```

### Environment

Current environment

```
#- PyTorch Lightning Version (e.g., 2.5.0):
#- PyTorch Version (e.g., 2.5):
#- Python version (e.g., 3.12):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
```

### More info

_No response_

cc @lantiga @mauvilsa

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/cli.py and the LightningCLI predict entry point, then trace how --ckpt_path=best reaches the checkpoint_connector. Reproduce with the provided config.yaml and compare trainer.callbacks configuration with best-checkpoint discovery. Done means the supplied ModelCheckpoint configuration is correctly handled, or the failure explains the unmet condition clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
cli, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.