tensorflow / tensorflow/datasets
Checksum Failure when Downloading Datasets from Seqio Mixture
@tomvdw is already working on this.
Since Mar 22, 2023.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
Short description
In a Google Research repository, several users are hitting an issue where they get checksum errors for TFDS datasets they need to download (e.g. see bottom of this thread), because the checksum has changed. It isn't clear to me how this is intended to be avoided.
Environment information
-
Operating System: Mac OS
-
Python version: 3.7
-
tensorflow-datasets/tfds-nightlyversion: 4.4.0.dev202108200109 -
tensorflow/tf-nightlyversion: 2.6.0 -
Does the issue still exists with the last
tfds-nightlypackage (pip install --upgrade tfds-nightly) ? -
Yes
Reproduction instructions
Clone this Google repository. Then in python flan/v2/run_example.py comment in line 86.
pip install --upgrade pip
pip install -r flan/v2/requirements.txt
PYTHONPATH=. python flan/v2/run_example.py
If you share a colab, make sure to update the permissions to share it.
Link to logs
ERROR:absl:Failed to load task 'aeslc_template_0to10_no_opt_x_shot' as part of mixture 'flan2022_submix'
Traceback (most recent call last):
File "/home/henry/flan/FLAN/flan/v2/run_example.py", line 93, in <module>
dataset = selected_mixture.get_dataset(
File "/home/henry/anaconda3/envs/palm/lib/python3.9/site-packages/seqio/dataset_providers.py", line 1730, in get_dataset
ds = task.get_dataset(
File "/home/henry/anaconda3/envs/palm/lib/python3.9/site-packages/seqio/dataset_providers.py", line 1375, in get_dataset
ds = source.get_dataset(
File "/home/henry/anaconda3/envs/palm/lib/python3.9/site-packages/seqio/experimental.py", line 369, in get_dataset
train_ds = _get_maybe_sharded_dataset(
File "/home/henry/anaconda3/envs/palm/lib/python3.9/site-packages/seqio/experimental.py", line 329, in _get_maybe_sharded_dataset
num_shards = len(self._original_source.list_shards(split_))
File "/home/henry/anaconda3/envs/palm/lib/python3.9/site-packages/seqio/dataset_providers.py", line 510, in list_shards
return [_get_filename(info) for info in self.tfds_dataset.files(split)]
File "/home/henry/anaconda3/envs/palm/lib/python3.9/site-packages/seqio/utils.py", line 159, in files
split_info = self.builder.info.splits[split]
File "/home/henry/anaconda3/envs/palm/lib/python3.9/site-packages/tensorflow_datasets/core/splits.py", line 391, in __getitem__
raise KeyError(
KeyError: "Trying to access `splits['train']` but `splits` is empty. This likely indicate the dataset has not been generated yet."
Expected behavior
To download TFDS dataset successfully.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.