huggingface / huggingface/datasets

FORCE_REDOWNLOAD does not work

Open
#2,904 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

## Describe the bug
With GenerateMode.FORCE_REDOWNLOAD, the documentation says
+------------------------------------+-----------+---------+
| | Downloads | Dataset |
+====================================+===========+=========+
| `REUSE_DATASET_IF_EXISTS` (default)| Reuse | Reuse |
+------------------------------------+-----------+---------+
| `REUSE_CACHE_IF_EXISTS` | Reuse | Fresh |
+------------------------------------+-----------+---------+
| `FORCE_REDOWNLOAD` | Fresh | Fresh |
+------------------------------------+-----------+---------+

However, the old dataset is loaded even when FORCE_REDOWNLOAD is chosen.

## Steps to reproduce the bug
```python

import pandas as pd
from datasets import load_dataset, GenerateMode
pd.DataFrame(range(5), columns=['numbers']).to_csv('/tmp/test.tsv.gz', index=False)
ee = load_dataset('csv', data_files=['/tmp/test.tsv.gz'], delimiter='\t', split='train', download_mode=GenerateMode.FORCE_REDOWNLOAD)
print(ee)
pd.DataFrame(range(10), columns=['numerals']).to_csv('/tmp/test.tsv.gz', index=False)
ee = load_dataset('csv', data_files=['/tmp/test.tsv.gz'], delimiter='\t', split='train', download_mode=GenerateMode.FORCE_REDOWNLOAD)
print(ee)

```

## Expected results
Dataset({
features: ['numbers'],
num_rows: 5
})
Dataset({
features: ['numerals'],
num_rows: 10
})

## Actual results
Dataset({
features: ['numbers'],
num_rows: 5
})
Dataset({
features: ['numbers'],
num_rows: 5
})

## Environment info

- `datasets` version: 1.8.0
- Platform: Linux-4.14.181-108.257.amzn1.x86_64-x86_64-with-glibc2.10
- Python version: 3.7.10
- PyArrow version: 3.0.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.