Lightning-AI / Lightning-AI/pytorch-lightning

Doing full validation on step 0

Open
#20,985 0 comments 6 reactions 0 assignees View on GitHub
feature trainer trainer: validate
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Description & Motivation

Hi,

I've been trying to do full validation on step 0 but everything I've tried has failed in some way. I am aware of [this stale issue](https://github.com/Lightning-AI/pytorch-lightning/issues/4170) but I could not re-open it so I created this one. Running full validation on step 0 is very useful for the cases where we want to finetune an already well-perfoming model.

These are the things I've tried:
* Using `trainer.validate()` #4948 fails with DDP because if you manually invoke the `validate` method, strange things happen with the dataloaders and the DDP checks fail afterwards.
* Setting `trainer.num_sanity_val_steps` to `-1` so it runs the sanity check on the full validation dataset also fails. Because during the sanity checking the loggers are not properly set up. I tried various versions where I attempted to manually set the loggers before the sanity checking and even forcing them to log, but those also failed and became very unnecessarily hacky.
* Tried temporarily setting `trainer.val_check_interval` to `1` to force the validation to happen at step 1 at least, but then setting it back to its original value did not take any effect and the trianer kept validating at every step.

I feel like this should be easier to do and maybe I'm missing something.

Thanks in advance.

### Pitch

Running validation at step 0 is important for many finetuning pipelines, and I think it should be easier to run it robustly on DDP without having to hack many things around the trainer pipeline.

### Alternatives

_No response_

### Additional context

_No response_

cc @lantiga @borda @justusschock

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by reproducing full validation at step 0 under DDP, then trace the trainer.validate(), num_sanity_val_steps, and val_check_interval paths, including logger and dataloader setup. Done means a documented, robust way to validate at step 0 without the reported DDP, logging, or scheduling failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.