huggingface / huggingface/datasets
datasets.DownloadMode.FORCE_REDOWNLOAD use cache to download dataset features with load_dataset function
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
I have updated my dataset by adding a new feature, and push it to the hub. When I want to download it on my machine which contain the old version by using `datasets.load_dataset("your_dataset_name", download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)` I get an error (paste bellow).
Seems that the load_dataset function still use the old features schema instead of downloading everything new from the HUB.
I find a way to go around this issue by manually deleting the old dataset cache. But from my understanding of `datasets.DownloadMode.FORCE_REDOWNLOAD` option, the dataset cache should be ignored.
### Steps to reproduce the bug
1. Download your dataset in your machine using `datasets.load_dataset`
2. Create a new feature in your dataset and push it to the hub
3. On the same machine redownload your dataset using `datasets.load_dataset("your_dataset_name", download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)`
### Expected behavior
`
ValueError: Couldn't cast
id: string
level: string
context: list
child 0, element: string
type: string
answer: string
question: string
supporting_facts: list
child 0, element: string
fra_answer: string
fra_question: string
-- schema metadata --
huggingface: '{"info": {"features": {"id": {"dtype": "string", "_type": "' + 490
to
{'id': Value(dtype='string', id=None), 'level': Value(dtype='string', id=None), 'context': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'type': Value(dtype='string', id=None), 'answer': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'supporting_facts': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
because column names don't match
The above exception was the direct cause of the following exception:
DatasetGenerationError
...
DatasetGenerationError: An error occurred while generating the dataset`
### Environment info
datasets-2.16.1 huggingface-hub-0.20.2
Contributor guide
Assessment
This issue has not been assessed yet.