tensorflow / tensorflow/datasets

Nonsensical DL size in progress bar when some files are already downloaded

Open
#3,759 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug contributions welcome
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

Short description
Running tfds build <dataset> when some of the files have already been downloaded and some are missing results in meaningless sizes being displayed in the progress bar.

Environment information

  • Operating System: Ubuntu 18.04

  • Python version: 3.7.5

  • tensorflow-datasets/tfds-nightly version: tensorflow-datasets 4.5.2

  • tensorflow/tf-nightly version: tensorflow 2.7.0

  • Does the issue still exists with the last tfds-nightly package (pip install --upgrade tfds-nightly) ? Yes

Reproduction instructions

First, run tfds build voc/2012 and wait for it to finish.

Then, remove $TFDS_DATA_DIR/downloads/pjredd.com_media_files_VOCtra_11-May-20124U92MnDPGT0LX3SxafRBV6Swxu-nCPTdD_eO5pF2O8s.tar and $TFDS_DATA_DIR/voc/2012/4.0.0.

Then run tfds build voc/2012 again. The progress bar will show something like this:

Dl Size...: 100%|█████████████████████████████████████████████▉| 1850626566/1850628467 [00:12<00:00, 146829508.09 MiB/s]

This doesn't make sense, as it implies that the missing file is less than 1% of the total size of the dataset's files, even though it's almost 2 GB in size.

Link to logs
N/A

Expected behavior
The progress bar should display numbers that are correctly proportioned.

Additional context
This most likely happens because the code that reports progress is inconsistent about the units it uses. Here and here it uses bytes, while here it uses megabytes.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in tensorflow_datasets/core/download/downloader.py, especially the progress-reporting points around lines 144, 182, and 236. Reproduce the issue by building voc/2012, removing the specified archive and generated dataset, then rebuilding. Done means the progress bar shows correctly proportioned download sizes when some files are already present.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
data, tooling
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.