huggingface / huggingface/datasets
FORCE_REDOWNLOAD does not work
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
## Describe the bug
With GenerateMode.FORCE_REDOWNLOAD, the documentation says
+------------------------------------+-----------+---------+
| | Downloads | Dataset |
+====================================+===========+=========+
| `REUSE_DATASET_IF_EXISTS` (default)| Reuse | Reuse |
+------------------------------------+-----------+---------+
| `REUSE_CACHE_IF_EXISTS` | Reuse | Fresh |
+------------------------------------+-----------+---------+
| `FORCE_REDOWNLOAD` | Fresh | Fresh |
+------------------------------------+-----------+---------+
However, the old dataset is loaded even when FORCE_REDOWNLOAD is chosen.
## Steps to reproduce the bug
```python
import pandas as pd
from datasets import load_dataset, GenerateMode
pd.DataFrame(range(5), columns=['numbers']).to_csv('/tmp/test.tsv.gz', index=False)
ee = load_dataset('csv', data_files=['/tmp/test.tsv.gz'], delimiter='\t', split='train', download_mode=GenerateMode.FORCE_REDOWNLOAD)
print(ee)
pd.DataFrame(range(10), columns=['numerals']).to_csv('/tmp/test.tsv.gz', index=False)
ee = load_dataset('csv', data_files=['/tmp/test.tsv.gz'], delimiter='\t', split='train', download_mode=GenerateMode.FORCE_REDOWNLOAD)
print(ee)
```
## Expected results
Dataset({
features: ['numbers'],
num_rows: 5
})
Dataset({
features: ['numerals'],
num_rows: 10
})
## Actual results
Dataset({
features: ['numbers'],
num_rows: 5
})
Dataset({
features: ['numbers'],
num_rows: 5
})
## Environment info
- `datasets` version: 1.8.0
- Platform: Linux-4.14.181-108.257.amzn1.x86_64-x86_64-with-glibc2.10
- Python version: 3.7.10
- PyArrow version: 3.0.0
Contributor guide
Assessment
This issue has not been assessed yet.