tensorflow / tensorflow/datasets
Adding an option for updating cached community dataset script
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
Is your feature request related to a problem? Please describe.
When using community datasets, TFDS downloads the dataset script and uses the same cached script for further use. However, if there was an update/improvement in the original script, there is no option to use the latest script as TFDS uses the older cached version.
For example, in issue #3170, although the original script has been fixed, TFDS still gives an error when using
tfds.load('huggingface:xnli') as it is still using the old cache.
Describe the solution you'd like
We should add a ignore_cache/update_cache argument to tfds.load() function, specifically for community datasets, so that the user can download and use the latest dataset generation script when required.
Describe alternatives you've considered
The only workaround for this is for the user to manually search the generated dataset on their system and delete it so that TFDS would have to download the script and generate the dataset from scratch. However, this is not a very optimal solution.
Additional context
@Conchylicultor @vijayphoenix It would be great if you could have a look. Please let me know your thoughts on this. I would like to take this issue forward. Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the tfds.load entry point and reproduce the cached community-dataset behavior described with huggingface:xnli and issue #3170. Trace how community dataset scripts are cached, then verify that an ignore_cache or update_cache option fetches the latest script without requiring manual deletion of the generated dataset.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python, tensorflow
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100