Lightning-AI / Lightning-AI/pytorch-lightning

Separate validation precision in trainer arguments

Open
#20,606 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Description & Motivation

The trainer should have a separate precision parameter for training and validation.

The feature would allow trading validation speed for some loss of measurement exactness. This is useful when validation takes a noticeable fraction of the overall training time. This scenario is common when training language models (e.g., language translators), where the model quality is measured by comparing *autoregressively generated* outputs with reference sequences.

### Pitch

The functionality would be exposed via `precision_val` parameter:

```python
trainer = lightning.Trainer(
precision="16-mixed",
precision_val="16-true"
)
```

### Alternatives

The user might implement custom precision logic in `on_validation_start` and `on_validation_end` callbacks in their `LightningModule`.

However, `Lightning` seems to generally advise against using `.to(device, dtype)` to manipulate model/tensors representation in memory because `Lightning` itself handles that, so writing custom precision logic in `LightningModule` appears out-of-place and error-prone.

### Additional context

For inspiration, `Seq2SeqTrainer` in the `transformers` library supports the proposed functionality via `fp16_full_eval=True` and `bf16_full_eval=True` parameters.

Relevant docs: https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.Seq2SeqTrainingArguments.fp16_full_eval

cc @lantiga @borda

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how the Lightning Trainer currently handles the precision parameter across training and validation. Compare the proposed behavior with Transformers' Seq2SeqTrainer options and the documented on_validation_start/on_validation_end callback alternative; done means a separate precision_val setting controls validation without changing training precision.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.