huggingface / huggingface/datasets
IterableDataset with_format does not support 'device' keyword for jax
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
As seen here: https://huggingface.co/docs/datasets/use_with_jax dataset.with_format() supports the keyword 'device', to put data on a specific device when loaded as jax. However, when called on an IterableDataset, I got the error `TypeError: with_format() got an unexpected keyword argument 'device'`
Looking over the code, it seems IterableDataset support only pytorch and no support for jax device keyword?
https://github.com/huggingface/datasets/blob/fc5c84f36684343bff3e424cb0fd1ac5ecdd66da/src/datasets/iterable_dataset.py#L1029
### Steps to reproduce the bug
1. Load an IterableDataset (tested in streaming mode)
2. Call with_format('jax',device=device)
### Expected behavior
I expect to call `with_format('jax', device=device)` as per [documentation](https://huggingface.co/docs/datasets/use_with_jax) without error
### Environment info
Tested with installing newest (dev) and also pip release (2.10.1).
- `datasets` version: 2.10.2.dev0
- Platform: Linux-5.15.89+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- Huggingface_hub version: 0.12.1
- PyArrow version: 11.0.0
- Pandas version: 1.3.5
Contributor guide
Assessment
This issue has not been assessed yet.