Lightning-AI / Lightning-AI/pytorch-lightning

`trainer.predict` on ` IterableDataset`s leads to a warning

Open
#15,284 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

trainer: predict ver: 1.7.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

I am able to train and use my PyTorch Lightning module as expected, but =when using the `trainer.predict` method on a testing DataLoader of the same type as the training and validation ones, I am getting this warning raised:

```
Lightning couldn't infer the indices fetched for your dataloader
```

Looking through the code, it looks like this is raised in the prediction epoch loop when the `batch_sampler` is not of type `IndexBatchSamplerWrapper`:
https://github.com/Lightning-AI/lightning/blob/a9142d637a63bb21d5bab67a5cf0c397b4278169/src/pytorch_lightning/loops/epoch/prediction_epoch_loop.py#L170-L173

It looks like in a data utility function, this happens to every DataLoader by default...
https://github.com/Lightning-AI/lightning/blob/a9142d637a63bb21d5bab67a5cf0c397b4278169/src/pytorch_lightning/utilities/data.py#L350-L351

BUT, it is disabled for datasets of type `torch.utils.data.IterableDataset`, which my dataset is.
https://github.com/Lightning-AI/lightning/blob/a9142d637a63bb21d5bab67a5cf0c397b4278169/src/pytorch_lightning/utilities/data.py#L216-L220

This means that I cannot use `trainer.predict` with this dataset without having the warning raised by PyTorch Lightning.

### How to reproduce the bug

```python
1. Create a testing DataLoader with type `torch.utils.data.IterableDataset`.
2. Using a PyTorch Lightning model + trainer combo, call the method `trainer.predict(model=SOME_PL_MODEL, dataloaders=test_dataloader)`.
```

### Error messages and logs

```
.../python3.8/site-packages/pytorch_lightning/loops/epoch/prediction_epoch_loop.py:174: UserWarning: Lightning couldn't infer the indices fetched for your dataloader.
warning_cache.warn("Lightning couldn't infer the indices fetched for your dataloader.")
```

### Environment

Using `Trainer` and `LightningModule` here.

```
* CUDA:
- GPU:
- Tesla V100-SXM2-16GB
- available: True
- version: 11.1
* Lightning:
- pytorch-lightning: 1.7.7
- torch: 1.10.2+cu111
- torch-intermediate-layer-getter: 0.1.post1
- torchmetrics: 0.10.0
- torchvision: 0.11.3+cu111
* Packages:
- absl-py: 0.11.0
- aiohttp: 3.8.3
- aiosignal: 1.2.0
- alembic: 1.8.1
- altair: 4.2.0
- appdirs: 1.4.4
- argon2-cffi: 20.1.0
- astor: 0.8.1
- astroid: 2.12.12
- astunparse: 1.6.3
- async-generator: 1.10
- async-timeout: 4.0.2
- attrs: 20.3.0
- backcall: 0.2.0
- backports.zoneinfo: 0.2.1
- bcrypt: 3.2.0
- bertopic: 0.12.0
- bidict: 0.21.4
- bleach: 3.3.0
- blinker: 1.5
- blis: 0.7.4
- boto3: 1.16.7
- botocore: 1.19.7
- cachetools: 4.2.4
- catalogue: 2.0.6
- certifi: 2020.12.5
- cffi: 1.14.5
- chardet: 4.0.0
- charset-normalizer: 2.1.1
- click: 7.1.2
- cloudpickle: 1.6.0
- cmdstanpy: 0.9.68
- commonmark: 0.9.1
- configparser: 5.0.1
- convertdate: 2.3.2
- cryptography: 3.4.7
- cycler: 0.10.0
- cymem: 2.0.5
- cython: 0.29.23
- databricks-automl-runtime: 0.2.6.4
- databricks-cli: 0.16.3
- datasci: 0.0.3
- datasets: 2.6.1
- dbl-tempo: 0.1.2
- dbus-python: 1.2.16
- decorator: 5.0.6
- defusedxml: 0.7.1
- dgl-cu111: 0.6.1
- dill: 0.3.5.1
- diskcache: 5.2.1
- distlib: 0.3.4
- distro: 1.4.0
- distro-info: 0.23ubuntu1
- docker: 6.0.0
- docker-pycreds: 0.4.0
- entrypoints: 0.3
- ephem: 4.1.3
- facets-overview: 1.0.0
- fasttext: 0.9.2
- filelock: 3.0.12
- findspark: 1.3.0
- flake8: 5.0.4
- flask: 1.1.2
- flatbuffers: 2.0
- frozenlist: 1.3.1
- fsspec: 2022.10.0
- future: 0.18.2
- gast: 0.4.0
- gitdb: 4.0.7
- gitpython: 3.1.12
- google-auth: 1.22.1
- google-auth-oauthlib: 0.4.2
- google-pasta: 0.2.0
- greenlet: 1.1.3.post0
- grpcio: 1.39.0
- gunicorn: 20.0.4
- gviz-api: 1.10.0
- h5py: 3.1.0
- hdbscan: 0.8.28
- hijri-converter: 2.2.3
- holidays: 0.12
- horovod: 0.23.0
- htmlmin: 0.1.12
- huggingface-hub: 0.10.1
- hvac: 1.0.2
- idna: 2.10
- imagehash: 4.2.1
- imbalanced-learn: 0.8.1
- importlib-metadata: 3.10.0
- importlib-resources: 5.10.0
- iniconfig: 1.1.1
- ipykernel: 5.3.4
- ipython: 7.22.0
- ipython-genutils: 0.2.0
- ipywidgets: 7.6.3
- isodate: 0.6.0
- isort: 5.10.1
- itsdangerous: 1.1.0
- jedi: 0.17.2
- jinja2: 2.11.3
- jmespath: 0.10.0
- joblib: 1.0.1
- joblibspark: 0.3.0
- json-log-formatter: 0.3.0
- jsonschema: 3.2.0
- jupyter-client: 6.1.12
- jupyter-core: 4.7.1
- jupyterlab-pygments: 0.1.2
- jupyterlab-widgets: 1.0.0
- keras: 2.8.0
- keras-preprocessing: 1.1.2
- kiwisolver: 1.3.1
- koalas: 1.8.2
- korean-lunar-calendar: 0.2.1
- langcodes: 3.3.0
- lazy-object-proxy: 1.7.1
- libclang: 13.0.0
- lightgbm: 3.3.2
- llvmlite: 0.38.0
- lunarcalendar: 0.0.9
- mako: 1.1.3
- markdown: 3.3.3
- markupsafe: 2.0.1
- matplotlib: 3.4.2
- mccabe: 0.7.0
- missingno: 0.5.1
- mistune: 0.8.4
- mleap: 0.18.1
- mlflow: 1.30.0
- mlflow-skinny: 1.24.0
- multidict: 6.0.2
- multimethod: 1.7
- multiprocess: 0.70.13
- murmurhash: 1.0.5
- nbclient: 0.5.3
- nbconvert: 6.0.7
- nbformat: 5.1.3
- nest-asyncio: 1.5.1
- networkx: 2.5
- nltk: 3.6.1
- notebook: 6.3.0
- numba: 0.55.1
- numpy: 1.20.1
- oauthlib: 3.1.0
- opt-einsum: 3.3.0
- packaging: 21.3
- pandas: 1.2.4
- pandas-profiling: 3.1.0
- pandocfilters: 1.4.3
- paramiko: 2.7.2
- parso: 0.7.0
- pathtools: 0.1.2
- pathy: 0.6.0
- patsy: 0.5.1
- petastorm: 0.11.4
- pexpect: 4.8.0
- phik: 0.12.0
- pickleshare: 0.7.5
- pillow: 8.2.0
- pip: 21.0.1
- platformdirs: 2.5.2
- plotly: 5.5.0
- pluggy: 1.0.0
- pmdarima: 1.8.4
- preshed: 3.0.5
- prometheus-client: 0.15.0
- prometheus-flask-exporter: 0.20.3
- promise: 2.3
- prompt-toolkit: 3.0.17
- prophet: 1.0.1
- protobuf: 3.17.2
- psutil: 5.8.0
- psycopg2: 2.8.5
- ptyprocess: 0.7.0
- py: 1.11.0
- py4j: 0.10.9
- pyarrow: 9.0.0
- pyasn1: 0.4.8
- pyasn1-modules: 0.2.8
- pybind11: 2.9.1
- pycodestyle: 2.9.1
- pycparser: 2.20
- pydantic: 1.8.2
- pydeck: 0.8.0b4
- pydeprecate: 0.3.2
- pyflakes: 2.5.0
- pygments: 2.8.1
- pygobject: 3.36.0
- pyhcl: 0.4.4
- pylint: 2.15.5
- pymeeus: 0.5.11
- pympler: 1.0.1
- pynacl: 1.4.0
- pynndescent: 0.5.7
- pynvml: 11.4.1
- pyodbc: 4.0.30
- pyparsing: 2.4.7
- pyrsistent: 0.17.3
- pyspark: 3.0.1
- pystan: 2.19.1.1
- pytest: 7.1.3
- python-apt: 2.0.0+ubuntu0.20.4.8
- python-dateutil: 2.8.1
- python-editor: 1.0.4
- python-engineio: 4.3.0
- python-socketio: 5.4.1
- pytorch-lightning: 1.7.7
- pytz: 2020.5
- pytz-deprecation-shim: 0.1.0.post0
- pywavelets: 1.1.1
- pyyaml: 5.4.1
- pyzmq: 20.0.0
- querystring-parser: 1.2.4
- reckon: 1.1.1
- regex: 2021.4.4
- requests: 2.28.1
- requests-oauthlib: 1.3.0
- requests-unixsocket: 0.2.0
- responses: 0.18.0
- rich: 12.6.0
- rsa: 4.7.2
- s3transfer: 0.3.7
- sacremoses: 0.0.46
- sailthru-client: 2.3.5
- scikit-learn: 1.1.2
- scipy: 1.6.2
- seaborn: 0.11.1
- semver: 2.13.0
- send2trash: 1.5.0
- sentence-transformers: 2.2.2
- sentencepiece: 0.1.97
- sentry-sdk: 1.10.1
- seqeval: 1.2.2
- setproctitle: 1.3.2
- setuptools: 52.0.0
- setuptools-git: 1.2
- shap: 0.40.0
- shortuuid: 1.0.9
- simplejson: 3.17.2
- simpletransformers: 0.63.9
- six: 1.15.0
- slicer: 0.0.7
- smart-open: 5.2.0
- smmap: 3.0.5
- spacy: 3.2.1
- spacy-legacy: 3.0.8
- spacy-loggers: 1.0.1
- spark-tensorflow-distributor: 1.0.0
- sqlalchemy: 1.4.42
- sqlparse: 0.4.1
- srsly: 2.4.1
- ssh-import-id: 5.10
- statsmodels: 0.12.2
- streamlit: 1.13.0
- tabulate: 0.8.7
- tangled-up-in-unicode: 0.1.0
- tenacity: 6.2.0
- tensorboard: 2.10.1
- tensorboard-data-server: 0.6.1
- tensorboard-plugin-profile: 2.5.0
- tensorboard-plugin-wit: 1.8.1
- tensorflow: 2.8.0
- tensorflow-estimator: 2.8.0
- tensorflow-io-gcs-filesystem: 0.24.0
- termcolor: 1.1.0
- terminado: 0.9.4
- testpath: 0.4.4
- tf-estimator-nightly: 2.8.0.dev2021122109
- thinc: 8.0.12
- threadpoolctl: 2.1.0
- tokenizers: 0.10.3
- toml: 0.10.2
- tomli: 2.0.1
- tomlkit: 0.11.5
- toolz: 0.12.0
- torch: 1.10.2+cu111
- torch-intermediate-layer-getter: 0.1.post1
- torchmetrics: 0.10.0
- torchvision: 0.11.3+cu111
- tornado: 6.1
- tqdm: 4.64.1
- traitlets: 5.0.5
- transformers: 4.16.2
- typer: 0.3.2
- typing-extensions: 4.4.0
- tzdata: 2022.5
- tzlocal: 4.2
- ujson: 4.0.2
- umap-learn: 0.5.3
- unattended-upgrades: 0.1
- urllib3: 1.26.12
- validators: 0.20.0
- virtualenv: 20.4.1
- visions: 0.7.4
- wandb: 0.13.4
- wasabi: 0.8.2
- watchdog: 2.1.9
- wcwidth: 0.2.5
- webencodings: 0.5.1
- websocket-client: 0.57.0
- werkzeug: 1.0.1
- wheel: 0.36.2
- widgetsnbextension: 3.5.1
- wrapt: 1.12.1
- xgboost: 1.5.2
- xxhash: 3.1.0
- yarl: 1.8.1
- zipp: 3.4.1
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor: x86_64
- python: 3.8.10
- version: #92~18.04.1-Ubuntu SMP Wed Aug 31 17:21:08 UTC 2022
```

### More info

_No response_

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/pytorch_lightning/loops/epoch/prediction_epoch_loop.py around the warning and src/pytorch_lightning/utilities/data.py around the IterableDataset and batch sampler handling. Reproduce the warning with trainer.predict and an IterableDataset DataLoader, then trace how fetched indices are handled. Done means prediction no longer emits the reported warning for this dataset type, with the existing behavior preserved for other DataLoaders.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.