Lightning-AI / Lightning-AI/pytorch-lightning

Allow Using Specific Dataloaders in LightningCLI (Instead of a DataModule)

Open
#20,673 6 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

feature lightningcli
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Description & Motivation

I would like to configure the LightningCLI to use dataloaders in the `test` subcommand, similar to this example from [the docs](https://lightning.ai/docs/pytorch/stable/common/trainer.html#testing)
```python
trainer.test(dataloaders=test_dataloaders)
```

As far as I can tell this impossible using the LightningCLI because you can only pass a `datamodule` into it. My test set is defined differently from various training sets I have so this becomes annoying to create separate datamodules for each.

You can see that the `dataloaders` option is specifically ignored in [the `cli.py` `subcommands` method](https://github.com/Lightning-AI/pytorch-lightning/blob/df5dee674243e124a2bf34d9975dd586ff008d4b/src/lightning/pytorch/cli.py#L476).

### Pitch

I would like to specify `dataloaders` in the `LightningCLI`. Perhaps like:

```python
LightningCLI(
MyModule,
train_dataloaders=MyTrainDataloader,
test_dataloaders=MyTestDataloader,
args=args
)
```

### Alternatives

The alternative is having to create different datamodules for every combination of train/test dataloaders.
Or you have to simply just not use the CLI stuff and call lightning directly.

### Additional context

I would be willing to open a PR if this feature is deemed useful.

cc @lantiga @borda @mauvilsa

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/lightning/pytorch/cli.py at the subcommands method, where the issue says the dataloaders option is ignored, and compare the supported LightningCLI inputs with the trainer.test(dataloaders=...) usage in the linked documentation. Done means the CLI can configure test dataloaders without requiring a separate datamodule, with the requested train and test dataloader forms accounted for.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
cli, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.