Lightning-AI / Lightning-AI/pytorch-lightning

LightningModule.configure_callbacks overrides Trainer callbacks

Open
#18,784 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

discussion feature lightningmodule
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Outline & Motivation

In [TorchGeo](https://github.com/microsoft/torchgeo), we provide `LightningModules` for common tasks, like classification, regression, etc. We also provide default callbacks using `LightningModule.configure_callbacks` as suggested in #18480. However, it seems that these default callbacks override any user-specific callbacks provided to the `Trainer`. According to [the documentation](https://lightning.ai/docs/pytorch/stable/common/lightning_module.html#configure-callbacks), this actually seems to be the intended behavior:

> the list or a callback returned here will be merged with the list of callbacks passed to the Trainer’s `callbacks` argument. If a callback returned here has the same type as one or several callbacks already present in the Trainer’s callbacks list, it will take priority and replace them.

I propose reversing this behavior such that Trainer `callbacks` override `LightningModule.configure_callbacks` when both share the same type.

### Pitch

Users should be able to override callbacks hard-coded in the `LightningModule` with whatever they pass to the `Trainer`. Otherwise the only way to change something like the model checkpoint frequency is to subclass the `LightningModule`, which is unideal.

Alternatively, if there's another way to provide model-specific default callbacks, please let me know.

### Additional context

@robmarkcole and @roybenhayun reported several issues to us that stemmed from this behavior. If we can't solve this, we may have to remove all default callbacks.

@calebrob6

cc @borda @carmocca @justusschock @awaelchli

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the LightningModule.configure_callbacks entry point and the Trainer callbacks behavior described in the linked documentation. Reproduce how same-type callbacks are merged and replaced, then review the discussion and related reports before deciding the intended precedence. Done means the documented behavior and tests establish whether Trainer callbacks override module defaults, or clarify an alternative for model-specific defaults.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.