tensorflow / tensorflow/datasets

NonMatchingChecksumError for sun397

Open
#4,807 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

/!\ PLEASE INCLUDE THE FULL STACKTRACE AND CODE SNIPPET

Short description
When

Environment information

  • Operating System:

  • Python version: <3..8>

  • tensorflow version: <2.11.0>

  • tensorflow-datasets version: <4.8.3>

  • tf-nightly version: <4.8.3.dev202303190046>

  • issue still exists with the last tfds-nightly package

Reproduction instructions

import tensorflow_datasets as tfds

dataset_name = "sun397/tfds:4.*.*"
name = dataset_name.split(':')[0]
dataset_builder = tfds.builder(dataset_name, data_dir=f'data/vtab/{name}/tfrecord')
dataset_builder.download_and_prepare(
   download_dir=f'data/vtab/{name}/raw',
   download_config=tfds.download.DownloadConfig(
        extract_dir=f'data/vtab/{name}/extracted',
        # max_examples_per_split=0,
    )
)

Link to logs

Traceback (most recent call last):
  File "/data1/NOAH/data/vtab-source/task_adaptation/data/sun397.py", line 40, in __init__
    dataset_builder.download_and_prepare(
  ……
  File "/data1/anaconda3/envs/NOAH/lib/python3.8/site-packages/tensorflow_datasets/core/download/download_manager.py", line 807, in _validate_checksums
    raise NonMatchingChecksumError(msg)
tensorflow_datasets.core.download.download_manager.NonMatchingChecksumError: Artifact https://vision.princeton.edu/projects/2010/SUN/SUN397.tar.gz, downloaded to data/vtab/sun397/tfds/raw/visio.princ.edu_proje_2010_SUN_SUN39YI7MRZf0CsLgBE9BMqMt4EoWzt0_oF_tQ9O6vET_wAc.tar.gz.tmp.aa94e417322b4f408a5f328d52c7e55c/SUN397.tar.gz, has wrong checksum:
* Expected: UrlInfo(size=36.39 GiB, checksum='f404130965a7ad77bed5ececc71e720ab50f3cc1e1bb257257610d38f3b928ec', filename='SUN397.tar.gz')
* Got: UrlInfo(size=294.59 MiB, checksum='de22a1d93ed230dd799d54281fad5356d36453cb0f6330b08139f035cb9e780c', filename='SUN397.tar.gz')
To debug, see: https://www.tensorflow.org/datasets/overview#fixing_nonmatchingchecksumerror

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the sun397.py reproduction and the _validate_checksums entry point in tensorflow_datasets/core/download/download_manager.py. Run the provided download_and_prepare example and compare the expected and downloaded SUN397.tar.gz metadata. Done means the reported checksum mismatch is resolved for this dataset download and the reproduction completes successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.