tensorflow / tensorflow/datasets

Checksum Failure when Downloading Datasets from Seqio Mixture

Open
#4,804 4 comments 0 reactions 1 assignee View on GitHub

@tomvdw is already working on this.

Since Mar 22, 2023.

bug
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

Short description
In a Google Research repository, several users are hitting an issue where they get checksum errors for TFDS datasets they need to download (e.g. see bottom of this thread), because the checksum has changed. It isn't clear to me how this is intended to be avoided.

Environment information

  • Operating System: Mac OS

  • Python version: 3.7

  • tensorflow-datasets/tfds-nightly version: 4.4.0.dev202108200109

  • tensorflow/tf-nightly version: 2.6.0

  • Does the issue still exists with the last tfds-nightly package (pip install --upgrade tfds-nightly) ?

  • Yes

Reproduction instructions

Clone this Google repository. Then in python flan/v2/run_example.py comment in line 86.

pip install --upgrade pip
pip install -r flan/v2/requirements.txt
PYTHONPATH=. python flan/v2/run_example.py

If you share a colab, make sure to update the permissions to share it.

Link to logs

ERROR:absl:Failed to load task 'aeslc_template_0to10_no_opt_x_shot' as part of mixture 'flan2022_submix'
Traceback (most recent call last):
  File "/home/henry/flan/FLAN/flan/v2/run_example.py", line 93, in <module>
    dataset = selected_mixture.get_dataset(
  File "/home/henry/anaconda3/envs/palm/lib/python3.9/site-packages/seqio/dataset_providers.py", line 1730, in get_dataset
    ds = task.get_dataset(
  File "/home/henry/anaconda3/envs/palm/lib/python3.9/site-packages/seqio/dataset_providers.py", line 1375, in get_dataset
    ds = source.get_dataset(
  File "/home/henry/anaconda3/envs/palm/lib/python3.9/site-packages/seqio/experimental.py", line 369, in get_dataset
    train_ds = _get_maybe_sharded_dataset(
  File "/home/henry/anaconda3/envs/palm/lib/python3.9/site-packages/seqio/experimental.py", line 329, in _get_maybe_sharded_dataset
    num_shards = len(self._original_source.list_shards(split_))
  File "/home/henry/anaconda3/envs/palm/lib/python3.9/site-packages/seqio/dataset_providers.py", line 510, in list_shards
    return [_get_filename(info) for info in self.tfds_dataset.files(split)]
  File "/home/henry/anaconda3/envs/palm/lib/python3.9/site-packages/seqio/utils.py", line 159, in files
    split_info = self.builder.info.splits[split]
  File "/home/henry/anaconda3/envs/palm/lib/python3.9/site-packages/tensorflow_datasets/core/splits.py", line 391, in __getitem__
    raise KeyError(
KeyError: "Trying to access `splits['train']` but `splits` is empty. This likely indicate the dataset has not been generated yet."

Expected behavior
To download TFDS dataset successfully.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.