huggingface / huggingface/datasets
`from_list` fails while `from_generator` works for large datasets
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
I am constructing a large time series dataset and observed that first constructing a list of entries and then using `Dataset.from_list` led to a crash as the number of items became large. However, this is not a problem when using `Dataset.from_generator`.
### Steps to reproduce the bug
#### Snippet A (crashes)
```py
from tqdm.auto import tqdm
import numpy as np
import datasets
def data_generator():
for i in tqdm(range(10_000_000)):
length = np.random.randint(2048)
series = np.random.rand(length)
yield {"target": series, "item_id": str(i), "start": np.datetime64("2000", "ms")}
data_list = list(data_generator())
ds = datasets.Dataset.from_list(data_list)
```
The last line crashes with
```
ArrowInvalid: Value 2147483761 too large to fit in C integer type
```
#### Snippet B (works)
```py
from tqdm.auto import tqdm
import numpy as np
import datasets
def data_generator():
for i in tqdm(range(10_000_000)):
length = np.random.randint(2048)
series = np.random.rand(length)
yield {"target": series, "item_id": str(i), "start": np.datetime64("2000", "ms")}
ds = datasets.Dataset.from_generator(data_generator)
```
### Expected behavior
I expected both the approaches to work or to fail similarly.
### Environment info
```
- `datasets` version: 3.6.0
- Platform: Linux-6.8.0-1029-aws-x86_64-with-glibc2.35
- Python version: 3.11.11
- `huggingface_hub` version: 0.32.2
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2025.3.0
```
Contributor guide
Assessment
This issue has not been assessed yet.