Lightning-AI / Lightning-AI/pytorch-lightning

Mixing the order of `--config` and `fit` in LightningCLI can cause confusion

Open
#19,714 8 comments 0 reactions 1 assignee View on GitHub

@awaelchli is already working on this.

Since Mar 29, 2024.

bug feature lightningcli ver: 2.2.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

Bug description

If you launch with

python main.py --config ... fit 

instead of

python main.py fit --config ...

Then you end up with cryptic errors such as

No action for key "trainer.accelerator

See report on twitter:
https://x.com/4ndr3aR/status/1772676605837484054?s=20

image

This is because the LightningCLI parser is built after applying the fit stage is parsed, on only that can match the provided config file. In general, the order matters for jsonargparse for good reasons.

Is there something we can do to improve the error for the user, with a sanity check before parsing begins?

Repro example:

import sys

import torch
from lightning.pytorch import LightningModule
from torch.utils.data import DataLoader, Dataset

from lightning.pytorch.cli import LightningCLI


class RandomDataset(Dataset):
    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len


class BoringModel(LightningModule):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.layer(x)

    def training_step(self, batch, batch_idx):
        return self(batch).sum()

    def configure_optimizers(self):
        return torch.optim.SGD(self.layer.parameters(), lr=0.1)

    def train_dataloader(self):
        return DataLoader(RandomDataset(32, 64), batch_size=2)


sys.argv = ["bug_report_model.py", "--config", "config.yaml", "fit"]
cli = LightningCLI(BoringModel)

cc @borda @carmocca @mauvilsa

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.