Lightning-AI / Lightning-AI/pytorch-lightning
Cross-validation while still enabling CLI control
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Description & Motivation
Related issue:
https://github.com/Lightning-AI/pytorch-lightning/issues/20544
Hi Lightning team,
I saw the discussion in the issue above about supporting cross-validation in LightningCLI. I tried implementing a simple approach where the CLI is used as a reusable training template, and cross-validation is orchestrated outside the CLI.
The idea is to wrap `LightningCLI` inside a function (e.g. `cli_main`) and call it multiple times with different fold indices passed through CLI arguments.
For example:
```python
def cli_main(args):
cli = LightningCLI(
model_class=MyLightningModule,
datamodule_class=KFoldDataModule,
args=args,
run=False,
)
cli.trainer.fit(cli.model, cli.datamodule)
def arg_parse():
pass
other_args, lightning_args = arg_parse()
# cross-validation orchestration
for k in range(5):
cli_main(
lightning_args + [
f"--data.k={k}",
f"--trainer.logger.init_args.version=cv-fold{k}",
]
)
```
In this setup LightningCLI acts as a reusable training template and all configuration is still controlled through CLI arguments.
Since this pattern works quite naturally with the current CLI design and doesn’t require much extra code, I was wondering whether something along these lines could potentially be integrated into LightningCLI in the future.
Curious to hear your thoughts.
### Pitch
_No response_
### Alternatives
_No response_
### Additional context
_No response_
cc @lantiga @mauvilsa
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing related issue #20544 and the LightningCLI usage shown in this issue. Evaluate the proposed reusable cli_main pattern, repeated fold arguments, and logger versioning, then define what an integrated cross-validation interface should do and how completion would be validated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100