Lightning-AI / Lightning-AI/pytorch-lightning
`trainer.predict` on ` IterableDataset`s leads to a warning
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
I am able to train and use my PyTorch Lightning module as expected, but =when using the `trainer.predict` method on a testing DataLoader of the same type as the training and validation ones, I am getting this warning raised:
```
Lightning couldn't infer the indices fetched for your dataloader
```
Looking through the code, it looks like this is raised in the prediction epoch loop when the `batch_sampler` is not of type `IndexBatchSamplerWrapper`:
https://github.com/Lightning-AI/lightning/blob/a9142d637a63bb21d5bab67a5cf0c397b4278169/src/pytorch_lightning/loops/epoch/prediction_epoch_loop.py#L170-L173
It looks like in a data utility function, this happens to every DataLoader by default...
https://github.com/Lightning-AI/lightning/blob/a9142d637a63bb21d5bab67a5cf0c397b4278169/src/pytorch_lightning/utilities/data.py#L350-L351
BUT, it is disabled for datasets of type `torch.utils.data.IterableDataset`, which my dataset is.
https://github.com/Lightning-AI/lightning/blob/a9142d637a63bb21d5bab67a5cf0c397b4278169/src/pytorch_lightning/utilities/data.py#L216-L220
This means that I cannot use `trainer.predict` with this dataset without having the warning raised by PyTorch Lightning.
### How to reproduce the bug
```python
1. Create a testing DataLoader with type `torch.utils.data.IterableDataset`.
2. Using a PyTorch Lightning model + trainer combo, call the method `trainer.predict(model=SOME_PL_MODEL, dataloaders=test_dataloader)`.
```
### Error messages and logs
```
.../python3.8/site-packages/pytorch_lightning/loops/epoch/prediction_epoch_loop.py:174: UserWarning: Lightning couldn't infer the indices fetched for your dataloader.
warning_cache.warn("Lightning couldn't infer the indices fetched for your dataloader.")
```
### Environment
Using `Trainer` and `LightningModule` here.
```
* CUDA:
- GPU:
- Tesla V100-SXM2-16GB
- available: True
- version: 11.1
* Lightning:
- pytorch-lightning: 1.7.7
- torch: 1.10.2+cu111
- torch-intermediate-layer-getter: 0.1.post1
- torchmetrics: 0.10.0
- torchvision: 0.11.3+cu111
* Packages:
- absl-py: 0.11.0
- aiohttp: 3.8.3
- aiosignal: 1.2.0
- alembic: 1.8.1
- altair: 4.2.0
- appdirs: 1.4.4
- argon2-cffi: 20.1.0
- astor: 0.8.1
- astroid: 2.12.12
- astunparse: 1.6.3
- async-generator: 1.10
- async-timeout: 4.0.2
- attrs: 20.3.0
- backcall: 0.2.0
- backports.zoneinfo: 0.2.1
- bcrypt: 3.2.0
- bertopic: 0.12.0
- bidict: 0.21.4
- bleach: 3.3.0
- blinker: 1.5
- blis: 0.7.4
- boto3: 1.16.7
- botocore: 1.19.7
- cachetools: 4.2.4
- catalogue: 2.0.6
- certifi: 2020.12.5
- cffi: 1.14.5
- chardet: 4.0.0
- charset-normalizer: 2.1.1
- click: 7.1.2
- cloudpickle: 1.6.0
- cmdstanpy: 0.9.68
- commonmark: 0.9.1
- configparser: 5.0.1
- convertdate: 2.3.2
- cryptography: 3.4.7
- cycler: 0.10.0
- cymem: 2.0.5
- cython: 0.29.23
- databricks-automl-runtime: 0.2.6.4
- databricks-cli: 0.16.3
- datasci: 0.0.3
- datasets: 2.6.1
- dbl-tempo: 0.1.2
- dbus-python: 1.2.16
- decorator: 5.0.6
- defusedxml: 0.7.1
- dgl-cu111: 0.6.1
- dill: 0.3.5.1
- diskcache: 5.2.1
- distlib: 0.3.4
- distro: 1.4.0
- distro-info: 0.23ubuntu1
- docker: 6.0.0
- docker-pycreds: 0.4.0
- entrypoints: 0.3
- ephem: 4.1.3
- facets-overview: 1.0.0
- fasttext: 0.9.2
- filelock: 3.0.12
- findspark: 1.3.0
- flake8: 5.0.4
- flask: 1.1.2
- flatbuffers: 2.0
- frozenlist: 1.3.1
- fsspec: 2022.10.0
- future: 0.18.2
- gast: 0.4.0
- gitdb: 4.0.7
- gitpython: 3.1.12
- google-auth: 1.22.1
- google-auth-oauthlib: 0.4.2
- google-pasta: 0.2.0
- greenlet: 1.1.3.post0
- grpcio: 1.39.0
- gunicorn: 20.0.4
- gviz-api: 1.10.0
- h5py: 3.1.0
- hdbscan: 0.8.28
- hijri-converter: 2.2.3
- holidays: 0.12
- horovod: 0.23.0
- htmlmin: 0.1.12
- huggingface-hub: 0.10.1
- hvac: 1.0.2
- idna: 2.10
- imagehash: 4.2.1
- imbalanced-learn: 0.8.1
- importlib-metadata: 3.10.0
- importlib-resources: 5.10.0
- iniconfig: 1.1.1
- ipykernel: 5.3.4
- ipython: 7.22.0
- ipython-genutils: 0.2.0
- ipywidgets: 7.6.3
- isodate: 0.6.0
- isort: 5.10.1
- itsdangerous: 1.1.0
- jedi: 0.17.2
- jinja2: 2.11.3
- jmespath: 0.10.0
- joblib: 1.0.1
- joblibspark: 0.3.0
- json-log-formatter: 0.3.0
- jsonschema: 3.2.0
- jupyter-client: 6.1.12
- jupyter-core: 4.7.1
- jupyterlab-pygments: 0.1.2
- jupyterlab-widgets: 1.0.0
- keras: 2.8.0
- keras-preprocessing: 1.1.2
- kiwisolver: 1.3.1
- koalas: 1.8.2
- korean-lunar-calendar: 0.2.1
- langcodes: 3.3.0
- lazy-object-proxy: 1.7.1
- libclang: 13.0.0
- lightgbm: 3.3.2
- llvmlite: 0.38.0
- lunarcalendar: 0.0.9
- mako: 1.1.3
- markdown: 3.3.3
- markupsafe: 2.0.1
- matplotlib: 3.4.2
- mccabe: 0.7.0
- missingno: 0.5.1
- mistune: 0.8.4
- mleap: 0.18.1
- mlflow: 1.30.0
- mlflow-skinny: 1.24.0
- multidict: 6.0.2
- multimethod: 1.7
- multiprocess: 0.70.13
- murmurhash: 1.0.5
- nbclient: 0.5.3
- nbconvert: 6.0.7
- nbformat: 5.1.3
- nest-asyncio: 1.5.1
- networkx: 2.5
- nltk: 3.6.1
- notebook: 6.3.0
- numba: 0.55.1
- numpy: 1.20.1
- oauthlib: 3.1.0
- opt-einsum: 3.3.0
- packaging: 21.3
- pandas: 1.2.4
- pandas-profiling: 3.1.0
- pandocfilters: 1.4.3
- paramiko: 2.7.2
- parso: 0.7.0
- pathtools: 0.1.2
- pathy: 0.6.0
- patsy: 0.5.1
- petastorm: 0.11.4
- pexpect: 4.8.0
- phik: 0.12.0
- pickleshare: 0.7.5
- pillow: 8.2.0
- pip: 21.0.1
- platformdirs: 2.5.2
- plotly: 5.5.0
- pluggy: 1.0.0
- pmdarima: 1.8.4
- preshed: 3.0.5
- prometheus-client: 0.15.0
- prometheus-flask-exporter: 0.20.3
- promise: 2.3
- prompt-toolkit: 3.0.17
- prophet: 1.0.1
- protobuf: 3.17.2
- psutil: 5.8.0
- psycopg2: 2.8.5
- ptyprocess: 0.7.0
- py: 1.11.0
- py4j: 0.10.9
- pyarrow: 9.0.0
- pyasn1: 0.4.8
- pyasn1-modules: 0.2.8
- pybind11: 2.9.1
- pycodestyle: 2.9.1
- pycparser: 2.20
- pydantic: 1.8.2
- pydeck: 0.8.0b4
- pydeprecate: 0.3.2
- pyflakes: 2.5.0
- pygments: 2.8.1
- pygobject: 3.36.0
- pyhcl: 0.4.4
- pylint: 2.15.5
- pymeeus: 0.5.11
- pympler: 1.0.1
- pynacl: 1.4.0
- pynndescent: 0.5.7
- pynvml: 11.4.1
- pyodbc: 4.0.30
- pyparsing: 2.4.7
- pyrsistent: 0.17.3
- pyspark: 3.0.1
- pystan: 2.19.1.1
- pytest: 7.1.3
- python-apt: 2.0.0+ubuntu0.20.4.8
- python-dateutil: 2.8.1
- python-editor: 1.0.4
- python-engineio: 4.3.0
- python-socketio: 5.4.1
- pytorch-lightning: 1.7.7
- pytz: 2020.5
- pytz-deprecation-shim: 0.1.0.post0
- pywavelets: 1.1.1
- pyyaml: 5.4.1
- pyzmq: 20.0.0
- querystring-parser: 1.2.4
- reckon: 1.1.1
- regex: 2021.4.4
- requests: 2.28.1
- requests-oauthlib: 1.3.0
- requests-unixsocket: 0.2.0
- responses: 0.18.0
- rich: 12.6.0
- rsa: 4.7.2
- s3transfer: 0.3.7
- sacremoses: 0.0.46
- sailthru-client: 2.3.5
- scikit-learn: 1.1.2
- scipy: 1.6.2
- seaborn: 0.11.1
- semver: 2.13.0
- send2trash: 1.5.0
- sentence-transformers: 2.2.2
- sentencepiece: 0.1.97
- sentry-sdk: 1.10.1
- seqeval: 1.2.2
- setproctitle: 1.3.2
- setuptools: 52.0.0
- setuptools-git: 1.2
- shap: 0.40.0
- shortuuid: 1.0.9
- simplejson: 3.17.2
- simpletransformers: 0.63.9
- six: 1.15.0
- slicer: 0.0.7
- smart-open: 5.2.0
- smmap: 3.0.5
- spacy: 3.2.1
- spacy-legacy: 3.0.8
- spacy-loggers: 1.0.1
- spark-tensorflow-distributor: 1.0.0
- sqlalchemy: 1.4.42
- sqlparse: 0.4.1
- srsly: 2.4.1
- ssh-import-id: 5.10
- statsmodels: 0.12.2
- streamlit: 1.13.0
- tabulate: 0.8.7
- tangled-up-in-unicode: 0.1.0
- tenacity: 6.2.0
- tensorboard: 2.10.1
- tensorboard-data-server: 0.6.1
- tensorboard-plugin-profile: 2.5.0
- tensorboard-plugin-wit: 1.8.1
- tensorflow: 2.8.0
- tensorflow-estimator: 2.8.0
- tensorflow-io-gcs-filesystem: 0.24.0
- termcolor: 1.1.0
- terminado: 0.9.4
- testpath: 0.4.4
- tf-estimator-nightly: 2.8.0.dev2021122109
- thinc: 8.0.12
- threadpoolctl: 2.1.0
- tokenizers: 0.10.3
- toml: 0.10.2
- tomli: 2.0.1
- tomlkit: 0.11.5
- toolz: 0.12.0
- torch: 1.10.2+cu111
- torch-intermediate-layer-getter: 0.1.post1
- torchmetrics: 0.10.0
- torchvision: 0.11.3+cu111
- tornado: 6.1
- tqdm: 4.64.1
- traitlets: 5.0.5
- transformers: 4.16.2
- typer: 0.3.2
- typing-extensions: 4.4.0
- tzdata: 2022.5
- tzlocal: 4.2
- ujson: 4.0.2
- umap-learn: 0.5.3
- unattended-upgrades: 0.1
- urllib3: 1.26.12
- validators: 0.20.0
- virtualenv: 20.4.1
- visions: 0.7.4
- wandb: 0.13.4
- wasabi: 0.8.2
- watchdog: 2.1.9
- wcwidth: 0.2.5
- webencodings: 0.5.1
- websocket-client: 0.57.0
- werkzeug: 1.0.1
- wheel: 0.36.2
- widgetsnbextension: 3.5.1
- wrapt: 1.12.1
- xgboost: 1.5.2
- xxhash: 3.1.0
- yarl: 1.8.1
- zipp: 3.4.1
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor: x86_64
- python: 3.8.10
- version: #92~18.04.1-Ubuntu SMP Wed Aug 31 17:21:08 UTC 2022
```
### More info
_No response_
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/pytorch_lightning/loops/epoch/prediction_epoch_loop.py around the warning and src/pytorch_lightning/utilities/data.py around the IterableDataset and batch sampler handling. Reproduce the warning with trainer.predict and an IterableDataset DataLoader, then trace how fetched indices are handled. Done means prediction no longer emits the reported warning for this dataset type, with the existing behavior preserved for other DataLoaders.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100