huggingface / huggingface/datasets

python core dump when downloading dataset

Open
#7,879 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

When downloading a dataset in streamed mode and exiting the program before the download completes, the python program core dumps when exiting:

```
terminate called without an active exception
Aborted (core dumped)
```

Tested with python 3.12.3, python 3.9.21

### Steps to reproduce the bug

Create python venv:

```bash
python -m venv venv
./venv/bin/activate
pip install datasets==4.4.1
```

Execute the following program:

```
from datasets import load_dataset
ds = load_dataset("HuggingFaceFW/fineweb-2", 'hrv_Latn', split="test", streaming=True)
for sample in ds:
break
```

### Expected behavior

Clean program exit

### Environment info

described above

**note**: the example works correctly when using ```datasets==3.1.0```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.