pytorch / pytorch/vision

Error Downloading Kinetics 400 dataset

Open
#5,865 7 comments 0 reactions 1 assignee View on GitHub

@pmeier is already working on this.

Since Apr 26, 2022.

module: datasets
Dominant language
Python
Stars
17.9k
Forks
7.3k
Avg merge
1d 15h
Merged PRs (30d)
13

Description

🐛 Describe the bug

Error in downloading Kinetics 400 dataset. Download stops at training set tar file 121 (full k400 dataset is 200+ tar files).

I know this isn't a problem with the disk being out of write space. This has happened twice with different users and environments.

Not seeing this exact error raised as an issue anywhere else. Seems something is wrong with one of the downloaded files? Any ideas or workaround would be greatly appreciated!

Thank you!!!!!

Code to produce Error:

import torchvision
kinetics_path = '~/kinetics400'
kinetics400_dataset = torchvision.datasets.Kinetics(root=kinetics_path, 
                                                    frames_per_clip=5,
                                                    num_classes='400',
                                                    download=True)

Last part of error message (top of message is cut off, sorry.)

Traceback (most recent call last):
  File "/bsdir/donwload_kinetics.py", line 4, in <module>
    kinetics400_dataset = torchvision.datasets.Kinetics(root=kinetics_path,
  File "/bsdir/miniconda3/envs/torch_cuda_11_3/lib/python3.9/site-packages/torchvision/datasets/kinetics.py", line 127, in __init__
    self.download_and_process_videos()
  File "/bsdir/miniconda3/envs/torch_cuda_11_3/lib/python3.9/site-packages/torchvision/datasets/kinetics.py", line 152, in download_and_process_videos
    self._download_videos()
  File "/bsdir/miniconda3/envs/torch_cuda_11_3/lib/python3.9/site-packages/torchvision/datasets/kinetics.py", line 184, in _download_videos
    download_and_extract_archive(line, tar_path, self.split_folder)
  File "/bsdir/miniconda3/envs/torch_cuda_11_3/lib/python3.9/site-packages/torchvision/datasets/utils.py", line 431, in download_and_extract_archive
    extract_archive(archive, extract_root, remove_finished)
  File "/bsdir/miniconda3/envs/torch_cuda_11_3/lib/python3.9/site-packages/torchvision/datasets/utils.py", line 408, in extract_archive
    extractor(from_path, to_path, compression)
  File "/bsdir/miniconda3/envs/torch_cuda_11_3/lib/python3.9/site-packages/torchvision/datasets/utils.py", line 271, in _extract_tar
    with tarfile.open(from_path, f"r:{compression[1:]}" if compression else "r") as tar:
  File "/bsdir/miniconda3/envs/torch_cuda_11_3/lib/python3.9/tarfile.py", line 1629, in open
    return func(name, filemode, fileobj, **kwargs)
  File "/bsdir/miniconda3/envs/torch_cuda_11_3/lib/python3.9/tarfile.py", line 1686, in gzopen
    raise ReadError("not a gzip file")
tarfile.ReadError: not a gzip file
Versions

PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 16.04.7 LTS (x86_64)
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
Clang version: Could not collect
CMake version: version 3.5.1
Libc version: glibc-2.23

Python version: 3.9.7 (default, Sep 16 2021, 13:09:58) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.4.0-210-generic-x86_64-with-glibc2.23
Is CUDA available: True
CUDA runtime version: 9.0.176
GPU models and configuration:
GPU 0: TITAN Xp
GPU 1: TITAN Xp
GPU 2: TITAN Xp
GPU 3: TITAN Xp
GPU 4: TITAN Xp
GPU 5: TITAN Xp

Nvidia driver version: 455.45.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.5.1.10
/usr/lib/x86_64-linux-gnu/libcudnn.so.6.0.21
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.2
[pip3] numpydoc==1.2
[pip3] steerable-pytorch==0.1
[pip3] torch==1.11.0
[pip3] torchaudio==0.11.0
[pip3] torchvision==0.12.0
[conda] blas 1.0 mkl
[conda] cudatoolkit 11.3.1 h2bc3f7f_2
[conda] libblas 3.9.0 12_linux64_mkl conda-forge
[conda] libcblas 3.9.0 12_linux64_mkl conda-forge
[conda] liblapack 3.9.0 12_linux64_mkl conda-forge
[conda] liblapacke 3.9.0 12_linux64_mkl conda-forge
[conda] mkl 2021.4.0 h06a4308_640
[conda] mkl-service 2.4.0 py39h7f8727e_0
[conda] mkl_fft 1.3.1 py39hd3c417c_0
[conda] mkl_random 1.2.2 py39h51133e4_0
[conda] mypy_extensions 0.4.3 py39h06a4308_1
[conda] numpy 1.21.2 py39h20f2e39_0
[conda] numpy-base 1.21.2 py39h79a1101_0
[conda] numpydoc 1.2 pyhd3eb1b0_0
[conda] pytorch 1.11.0 py3.9_cuda11.3_cudnn8.2.0_0 pytorch
[conda] pytorch-mutex 1.0 cuda pytorch

cc @pmeier @YosuaMichael

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.