Lightning-AI / Lightning-AI/pytorch-lightning

Trainer.predict and Trainer.test reset model state to evaluation

Open
#21,410 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug ver: 2.5.x working as intended
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

I want to have dropout active in evaluation mode to generate random outputs. For this, I set the model to train mode before performing the prediction. However, using lightning's `Trainer.predict` resets the model to evaluation mode, essentially disabling dropout, which leads to deterministic outputs. Running the prediction on the raw model works as expected.

Note: Contrary to the version dropdown selection, I am running version 2.6.0.

### What version are you seeing the problem on?

master

### Reproduced in studio

_No response_

### How to reproduce the bug

```python
# %%
import pytorch_lightning as pl
import torch
from torch.utils.data import DataLoader, TensorDataset

# %%
class SimpleModel(pl.LightningModule):
def __init__(self):
super().__init__()
self.layer = torch.nn.Linear(10, 1)
self.dropout = torch.nn.Dropout(0.5)

def forward(self, x):
return self.dropout(self.layer(x))

def test_step(self, batch, batch_idx):
x, y = batch
out = self(x)
loss = torch.nn.functional.mse_loss(out, y)
self.log("test_loss", loss)
return loss

def predict_step(self, batch, batch_idx):
x, y = batch
return self(x)

# Data
X = torch.randn(100, 10)
y = torch.randn(100, 1)
ds = TensorDataset(X, y)
dl = DataLoader(ds, batch_size=10)

# Model
pl.seed_everything(42)
model = SimpleModel()
trainer = pl.Trainer(accelerator="cpu", devices=1)

# %%
print("--- Evaluate model using Trainer ---")
print("--- Run 1: train(False) ---")
model.train(False)
predictions = trainer.predict(model, dataloaders=dl)
y_pred_deterministic = torch.cat(predictions)

print("--- Run 2: train(True) ---")
model.train(True)
predictions = trainer.predict(model, dataloaders=dl)
y_pred_stochastic = torch.cat(predictions)

are_same = torch.allclose(y_pred_deterministic, y_pred_stochastic)
print(f"Predictions are the same: {are_same}")

print("------------------------------------")
# %%
print("--- Evaluate model using Raw Model Loop ---")
print("--- Run 3: raw train(True) ---")
model.train(False)
with torch.no_grad():
raw_predictions = [model(x) for x, _ in dl]
y_pred_raw_deterministic = torch.cat(raw_predictions)

print("--- Run 4: raw train(True) ---")
model.train(True)
with torch.no_grad():
raw_predictions = [model(x) for x, _ in dl]
y_pred_raw_stochastic = torch.cat(raw_predictions)

are_same = torch.allclose(y_pred_raw_deterministic, y_pred_raw_stochastic)
print(f"Raw predictions are the same: {are_same}")

# %%
```

### Error messages and logs

```
# Error messages and logs here please
```

### Environment

Current environment

* CUDA:
- GPU: None
- available: False
- version: 12.8
* Lightning:
- lightning-utilities: 0.15.2
- pytorch-lightning: 2.6.0
- torch: 2.9.1
- torchinfo: 1.8.0
- torchmetrics: 1.8.2
- torchview: 0.2.7
* Packages:
- absl-py: 2.3.1
- aiohappyeyeballs: 2.6.1
- aiohttp: 3.13.2
- aiosignal: 1.4.0
- alembic: 1.17.2
- alibi-detect: 0.13.0
- altair: 6.0.0
- annotated-doc: 0.0.4
- annotated-types: 0.7.0
- anyio: 4.12.0
- appdirs: 1.4.4
- argon2-cffi: 25.1.0
- argon2-cffi-bindings: 25.1.0
- arrow: 1.4.0
- astor: 0.8.1
- asttokens: 3.0.1
- astunparse: 1.6.3
- async-lru: 2.0.5
- attrs: 25.4.0
- autocommand: 2.2.2
- babel: 2.17.0
- backports-zstd: 1.2.0
- backports.tarfile: 1.2.0
- beautifulsoup4: 4.14.3
- bleach: 6.3.0
- blinker: 1.9.0
- bokeh: 3.8.1
- brotli: 1.2.0
- cachetools: 6.2.2
- cartes: 0.8.5
- cartopy: 0.25.0
- catalogue: 2.0.10
- certifi: 2025.11.12
- cffi: 2.0.0
- charset-normalizer: 3.4.4
- cheroot: 11.1.2
- click: 8.2.1
- cloudpickle: 3.1.2
- cmdstanpy: 1.3.0
- coloredlogs: 15.0.1
- comm: 0.2.3
- contourpy: 1.3.3
- cramjam: 2.11.0
- cryptography: 46.0.3
- cycler: 0.12.1
- databricks-sdk: 0.73.0
- debugpy: 1.8.17
- decorator: 5.2.1
- defusedxml: 0.7.1
- dill: 0.3.9
- dm-tree: 0.1.9
- docker: 7.1.0
- etils: 1.13.0
- executing: 2.2.1
- fastapi: 0.124.0
- fastjsonschema: 2.21.2
- fastparquet: 2024.11.0
- filelock: 3.20.0
- flask: 3.1.2
- flask-cors: 6.0.1
- flatbuffers: 25.9.23
- flexcache: 0.3
- flexparser: 0.4
- fonttools: 4.61.0
- fqdn: 1.5.1
- frozenlist: 1.8.0
- fsspec: 2025.12.0
- gast: 0.7.0
- gcsfs: 2025.12.0
- geopandas: 1.1.1
- gitdb: 4.0.12
- gitpython: 3.1.45
- google-api-core: 2.28.1
- google-auth: 2.43.0
- google-auth-oauthlib: 1.2.2
- google-cloud-core: 2.5.0
- google-cloud-storage: 3.6.0
- google-cloud-storage-control: 1.8.0
- google-crc32c: 1.7.1
- google-pasta: 0.2.0
- google-resumable-media: 2.8.0
- googleapis-common-protos: 1.72.0
- graphene: 3.4.3
- graphql-core: 3.2.7
- graphql-relay: 3.2.0
- graphviz: 0.21
- greenlet: 3.3.0
- grpc-google-iam-v1: 0.14.3
- grpcio: 1.76.0
- grpcio-status: 1.76.0
- gunicorn: 23.0.0
- gviz-api: 1.10.0
- h11: 0.16.0
- h2: 4.3.0
- h5py: 3.15.1
- hf-xet: 1.2.0
- holidays: 0.86
- hpack: 4.1.0
- httpcore: 1.0.9
- httpx: 0.28.1
- huggingface-hub: 0.36.0
- humanfriendly: 10.0
- hyperframe: 6.1.0
- idna: 3.11
- imageio: 2.37.2
- importlib-metadata: 8.7.0
- importlib-resources: 6.5.2
- impunity: 1.0.5
- inflate64: 1.0.4
- inflect: 7.3.1
- iniconfig: 2.3.0
- ipykernel: 7.1.0
- ipython: 9.8.0
- ipython-pygments-lexers: 1.1.1
- ipywidgets: 8.1.8
- isoduration: 20.11.0
- itsdangerous: 2.2.0
- jaraco-functools: 4.3.0
- jaraco.classes: 3.4.0
- jaraco.collections: 5.1.0
- jaraco.context: 6.0.1
- jaraco.functools: 4.0.1
- jaraco.text: 3.12.1
- jedi: 0.19.2
- jeepney: 0.9.0
- jinja2: 3.1.6
- joblib: 1.5.2
- json5: 0.12.1
- jsonpointer: 3.0.0
- jsonschema: 4.25.1
- jsonschema-specifications: 2025.9.1
- jupyter: 1.1.1
- jupyter-client: 8.6.3
- jupyter-console: 6.6.3
- jupyter-core: 5.9.1
- jupyter-events: 0.12.0
- jupyter-lsp: 2.3.0
- jupyter-server: 2.17.0
- jupyter-server-terminals: 0.5.3
- jupyterlab: 4.5.0
- jupyterlab-pygments: 0.3.0
- jupyterlab-server: 2.28.0
- jupyterlab-widgets: 3.0.16
- keopscore: 2.2.3
- keras: 3.12.0
- keras-tuner: 1.4.8
- keyring: 25.7.0
- kiwisolver: 1.4.9
- kt-legacy: 1.0.5
- lark: 1.3.1
- lazy-loader: 0.4
- libclang: 18.1.1
- librt: 0.7.3
- lightning-utilities: 0.15.2
- llvmlite: 0.46.0
- lxml: 6.0.2
- lz4: 4.4.5
- mako: 1.3.10
- markdown: 3.10
- markdown-it-py: 4.0.0
- markupsafe: 3.0.3
- matplotlib: 3.10.7
- matplotlib-inline: 0.2.1
- mdurl: 0.1.2
- metar: 1.11.0
- minio: 7.2.20
- mistune: 3.1.4
- ml-dtypes: 0.5.4
- mlflow: 3.5.1
- mlflow-skinny: 3.5.1
- mlflow-tracing: 3.5.1
- more-itertools: 10.8.0
- mpmath: 1.3.0
- msgpack: 1.1.2
- multidict: 6.7.0
- multivolumefile: 0.2.3
- mypy: 1.19.0
- mypy-extensions: 1.1.0
- namex: 0.1.0
- narwhals: 2.13.0
- nbclient: 0.10.2
- nbconvert: 7.16.6
- nbformat: 5.10.4
- nest-asyncio: 1.6.0
- networkx: 3.6.1
- notebook: 7.5.0
- notebook-shim: 0.2.4
- numba: 0.63.0
- numpy: 2.3.5
- nvidia-cublas-cu12: 12.8.4.1
- nvidia-cuda-cupti-cu12: 12.8.90
- nvidia-cuda-nvcc-cu12: 12.9.86
- nvidia-cuda-nvrtc-cu12: 12.8.93
- nvidia-cuda-runtime-cu12: 12.8.90
- nvidia-cudnn-cu12: 9.10.2.21
- nvidia-cufft-cu12: 11.3.3.83
- nvidia-cufile-cu12: 1.13.1.3
- nvidia-curand-cu12: 10.3.9.90
- nvidia-cusolver-cu12: 11.7.3.90
- nvidia-cusparse-cu12: 12.5.8.93
- nvidia-cusparselt-cu12: 0.7.1
- nvidia-nccl-cu12: 2.27.5
- nvidia-nvjitlink-cu12: 12.8.93
- nvidia-nvshmem-cu12: 3.3.20
- nvidia-nvtx-cu12: 12.8.90
- oauthlib: 3.3.1
- onnxruntime: 1.23.2
- openap: 2.4
- opencv-python: 4.11.0.86
- opentelemetry-api: 1.39.0
- opentelemetry-proto: 1.39.0
- opentelemetry-sdk: 1.39.0
- opentelemetry-semantic-conventions: 0.60b0
- opt-einsum: 3.4.0
- optree: 0.18.0
- orjson: 3.11.5
- overrides: 7.7.0
- packaging: 25.0
- pandas: 2.3.3
- pandocfilters: 1.5.1
- parso: 0.8.5
- pathspec: 0.12.1
- patsy: 1.0.2
- pexpect: 4.9.0
- pillow: 10.4.0
- pint: 0.25.2
- pitot: 0.3.2
- platformdirs: 4.5.1
- plotly: 6.5.0
- pluggy: 1.6.0
- prometheus-client: 0.23.1
- prompt-toolkit: 3.0.52
- propcache: 0.4.1
- properscoring: 0.1
- prophet: 1.2.1
- proto-plus: 1.26.1
- protobuf: 6.33.2
- psutil: 7.1.3
- ptyprocess: 0.7.0
- pure-eval: 0.2.3
- py7zr: 1.0.0
- pyarrow: 21.0.0
- pyasn1: 0.6.1
- pyasn1-modules: 0.4.2
- pybcj: 1.0.7
- pybind11: 3.0.1
- pycparser: 2.23
- pycryptodome: 3.23.0
- pycryptodomex: 3.23.0
- pydantic: 2.12.5
- pydantic-core: 2.41.5
- pygments: 2.19.2
- pyjwt: 2.10.1
- pykeops: 2.2.3
- pynverse: 0.1.4.6
- pyod: 2.0.6
- pyogrio: 0.12.1
- pyopensky: 2.15
- pyparsing: 3.2.5
- pyppmd: 1.2.0
- pyproj: 3.7.2
- pyshp: 3.0.3
- pytest: 9.0.2
- python-dateutil: 2.9.0.post0
- python-dotenv: 1.2.1
- python-json-logger: 4.0.0
- pytorch-lightning: 2.6.0
- pytz: 2025.2
- pyyaml: 6.0.3
- pyzmq: 27.1.0
- pyzstd: 0.19.0
- quantile-forest: 1.4.1
- ray: 2.52.1
- referencing: 0.37.0
- regex: 2025.11.3
- requests: 2.32.5
- requests-oauthlib: 2.0.0
- rfc3339-validator: 0.1.4
- rfc3986-validator: 0.1.1
- rfc3987-syntax: 1.1.0
- rich: 14.2.0
- rpds-py: 0.30.0
- rs1090: 0.4.14
- rsa: 4.9.1
- ruff: 0.14.8
- safetensors: 0.7.0
- scikit-image: 0.25.2
- scikit-learn: 1.7.2
- scipy: 1.16.3
- seaborn: 0.13.2
- secretstorage: 3.5.0
- send2trash: 1.8.3
- setuptools: 80.9.0
- sfoutils: 0.2.0
- shap: 0.50.0
- shapely: 2.1.2
- six: 1.17.0
- sklearn-quantile: 0.1.1
- slicer: 0.0.8
- smmap: 5.0.2
- soupsieve: 2.8
- sqlalchemy: 2.0.44
- sqlparse: 0.5.4
- stack-data: 0.6.3
- stanio: 0.5.1
- starlette: 0.50.0
- statsmodels: 0.14.6
- sympy: 1.14.0
- tensorboard: 2.20.0
- tensorboard-data-server: 0.7.2
- tensorboard-plugin-profile: 2.21.3
- tensorboardx: 2.6.4
- tensorflow: 2.20.0
- tensorflow-docs: 2025.12.2.70325
- tensorflow-probability: 0.25.0
- termcolor: 3.2.0
- terminado: 0.18.1
- texttable: 1.7.0
- tf-keras: 2.20.1
- threadpoolctl: 3.6.0
- tifffile: 2025.10.16
- tinycss2: 1.4.0
- tokenizers: 0.21.4
- toml: 0.10.2
- tomli: 2.0.1
- torch: 2.9.1
- torchinfo: 1.8.0
- torchmetrics: 1.8.2
- torchview: 0.2.7
- tornado: 6.5.2
- tqdm: 4.67.1
- traffic: 2.13.post16.dev0+9e00a83
- traitlets: 5.14.3
- transformers: 4.51.3
- trino: 0.336.0
- triton: 3.5.1
- tudcolors: 0.0.1
- typeguard: 4.3.0
- types-protobuf: 6.32.1.20251105
- types-requests: 2.32.4.20250913
- types-tensorflow: 2.18.0.20251008
- typing-extensions: 4.15.0
- typing-inspection: 0.4.2
- tzdata: 2025.2
- tzlocal: 5.3.1
- uri-template: 1.3.0
- urllib3: 2.6.1
- uvicorn: 0.38.0
- wcwidth: 0.2.14
- webcolors: 25.10.0
- webencodings: 0.5.1
- websocket-client: 1.9.0
- werkzeug: 3.1.4
- wheel: 0.45.1
- widgetsnbextension: 4.0.15
- wrapt: 2.0.1
- xprof: 2.21.3
- xyzservices: 2025.11.0
- yarl: 1.22.0
- zipp: 3.23.0
- zstandard: 0.25.0
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor:
- python: 3.13.9
- release: 6.17.9-200.fc42.x86_64
- version: #1 SMP PREEMPT_DYNAMIC Mon Nov 24 22:28:05 UTC 2025

### More info

_No response_

cc @ethanwharris

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Trainer.predict and Trainer.test entry points and run the provided Python reproduction against PyTorch Lightning 2.6.0. Trace where evaluation mode is applied during these calls; done means a model explicitly set to train mode can retain stochastic dropout behavior while predictions or tests run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.