tensorflow / tensorflow/datasets

Adding an option for updating cached community dataset script

Open
#3,172 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

Is your feature request related to a problem? Please describe.
When using community datasets, TFDS downloads the dataset script and uses the same cached script for further use. However, if there was an update/improvement in the original script, there is no option to use the latest script as TFDS uses the older cached version.
For example, in issue #3170, although the original script has been fixed, TFDS still gives an error when using
tfds.load('huggingface:xnli') as it is still using the old cache.

Describe the solution you'd like
We should add a ignore_cache/update_cache argument to tfds.load() function, specifically for community datasets, so that the user can download and use the latest dataset generation script when required.

Describe alternatives you've considered
The only workaround for this is for the user to manually search the generated dataset on their system and delete it so that TFDS would have to download the script and generate the dataset from scratch. However, this is not a very optimal solution.

Additional context
@Conchylicultor @vijayphoenix It would be great if you could have a look. Please let me know your thoughts on this. I would like to take this issue forward. Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the tfds.load entry point and reproduce the cached community-dataset behavior described with huggingface:xnli and issue #3170. Trace how community dataset scripts are cached, then verify that an ignore_cache or update_cache option fetches the latest script without requiring manual deletion of the generated dataset.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python, tensorflow
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.