Lightning-AI / Lightning-AI/pytorch-lightning
Doing full validation on step 0
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Description & Motivation
Hi,
I've been trying to do full validation on step 0 but everything I've tried has failed in some way. I am aware of [this stale issue](https://github.com/Lightning-AI/pytorch-lightning/issues/4170) but I could not re-open it so I created this one. Running full validation on step 0 is very useful for the cases where we want to finetune an already well-perfoming model.
These are the things I've tried:
* Using `trainer.validate()` #4948 fails with DDP because if you manually invoke the `validate` method, strange things happen with the dataloaders and the DDP checks fail afterwards.
* Setting `trainer.num_sanity_val_steps` to `-1` so it runs the sanity check on the full validation dataset also fails. Because during the sanity checking the loggers are not properly set up. I tried various versions where I attempted to manually set the loggers before the sanity checking and even forcing them to log, but those also failed and became very unnecessarily hacky.
* Tried temporarily setting `trainer.val_check_interval` to `1` to force the validation to happen at step 1 at least, but then setting it back to its original value did not take any effect and the trianer kept validating at every step.
I feel like this should be easier to do and maybe I'm missing something.
Thanks in advance.
### Pitch
Running validation at step 0 is important for many finetuning pipelines, and I think it should be easier to run it robustly on DDP without having to hack many things around the trainer pipeline.
### Alternatives
_No response_
### Additional context
_No response_
cc @lantiga @borda @justusschock
Contributor guide
Research direction
No files or tests are named. Start by reproducing full validation at step 0 under DDP, then trace the trainer.validate(), num_sanity_val_steps, and val_check_interval paths, including logger and dataloader setup. Done means a documented, robust way to validate at step 0 without the reported DDP, logging, or scheduling failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100