Lightning-AI / Lightning-AI/pytorch-lightning
Separate validation precision in trainer arguments
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Description & Motivation
The trainer should have a separate precision parameter for training and validation.
The feature would allow trading validation speed for some loss of measurement exactness. This is useful when validation takes a noticeable fraction of the overall training time. This scenario is common when training language models (e.g., language translators), where the model quality is measured by comparing *autoregressively generated* outputs with reference sequences.
### Pitch
The functionality would be exposed via `precision_val` parameter:
```python
trainer = lightning.Trainer(
precision="16-mixed",
precision_val="16-true"
)
```
### Alternatives
The user might implement custom precision logic in `on_validation_start` and `on_validation_end` callbacks in their `LightningModule`.
However, `Lightning` seems to generally advise against using `.to(device, dtype)` to manipulate model/tensors representation in memory because `Lightning` itself handles that, so writing custom precision logic in `LightningModule` appears out-of-place and error-prone.
### Additional context
For inspiration, `Seq2SeqTrainer` in the `transformers` library supports the proposed functionality via `fp16_full_eval=True` and `bf16_full_eval=True` parameters.
Relevant docs: https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.Seq2SeqTrainingArguments.fp16_full_eval
cc @lantiga @borda
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how the Lightning Trainer currently handles the precision parameter across training and validation. Compare the proposed behavior with Transformers' Seq2SeqTrainer options and the documented on_validation_start/on_validation_end callback alternative; done means a separate precision_val setting controls validation without changing training precision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100