huggingface / huggingface/datasets

`from_list` fails while `from_generator` works for large datasets

Open
#7,619 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

I am constructing a large time series dataset and observed that first constructing a list of entries and then using `Dataset.from_list` led to a crash as the number of items became large. However, this is not a problem when using `Dataset.from_generator`.

### Steps to reproduce the bug

#### Snippet A (crashes)

```py
from tqdm.auto import tqdm
import numpy as np
import datasets

def data_generator():
for i in tqdm(range(10_000_000)):
length = np.random.randint(2048)
series = np.random.rand(length)
yield {"target": series, "item_id": str(i), "start": np.datetime64("2000", "ms")}

data_list = list(data_generator())
ds = datasets.Dataset.from_list(data_list)
```
The last line crashes with
```
ArrowInvalid: Value 2147483761 too large to fit in C integer type
```

#### Snippet B (works)

```py
from tqdm.auto import tqdm
import numpy as np
import datasets

def data_generator():
for i in tqdm(range(10_000_000)):
length = np.random.randint(2048)
series = np.random.rand(length)
yield {"target": series, "item_id": str(i), "start": np.datetime64("2000", "ms")}

ds = datasets.Dataset.from_generator(data_generator)
```

### Expected behavior

I expected both the approaches to work or to fail similarly.

### Environment info

```
- `datasets` version: 3.6.0
- Platform: Linux-6.8.0-1029-aws-x86_64-with-glibc2.35
- Python version: 3.11.11
- `huggingface_hub` version: 0.32.2
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2025.3.0
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.