tensorflow / tensorflow/datasets

Cannot download_and_prepare Lm1b dataset

Open
#3,992 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

/!\ PLEASE INCLUDE THE FULL STACKTRACE AND CODE SNIPPET

Short description
The host website of Lm1b dataset is down, so downloading Lm1b dataset returns a 404 error. I have managed to download the .gz file of Lm1b dataset, however I found it hard to prepare Lm1b with a pre-downloaded gz file since I cannot call prepare() separately. Is there a way to find a Lm1b dataset mirror or to load Lm1b dataset from .gz file?

Environment information

  • Operating System: Ubuntu 18.04

  • Python version: 3.8.13

  • tensorflow-datasets/tfds-nightly tensorflow-datasets 4.4.0

  • tensorflow/tf-nightly version: tensorflow 2.8.1

  • Does the issue still exists with the last tfds-nightly package (pip install --upgrade tfds-nightly) ?
    yes

Reproduction instructions

import tensorflow_datasets as tfds
builder = tfds.text.Lm1b()
builder.download_and_prepare()

If you share a colab, make sure to update the permissions to share it.

Link to logs
Downloading and preparing dataset 1.67 GiB (download: 1.67 GiB, generated: 4.40 GiB, total: 6.07 GiB) to /home/marstan/tensorflow_datasets/lm1b/1.1.0...
Extraction completed...: 0 file [00:01, ? file/s] | 0/1 [00:00<?, ? url/s]
Dl Size...: 0 MiB [00:01, ? MiB/s]
Dl Completed...: 0%| | 0/1 [00:01<?, ? url/s]
Traceback (most recent call last):
File "", line 1, in
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow_datasets/core/dataset_builder.py", line 439, in download_and_prepare
self._download_and_prepare(
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow_datasets/core/dataset_builder.py", line 1113, in _download_and_prepare
split_generators = self._split_generators( # pylint: disable=unexpected-keyword-arg
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow_datasets/text/lm1b.py", line 87, in _split_generators
lm1b_path = dl_manager.download_and_extract(_DOWNLOAD_URL)
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow_datasets/core/download/download_manager.py", line 634, in download_and_extract
return _map_promise(self._download_extract, url_or_urls)
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow_datasets/core/download/download_manager.py", line 767, in _map_promise
res = tf.nest.map_structure(lambda p: p.get(), all_promises) # Wait promises
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow/python/util/nest.py", line 914, in map_structure
structure[0], [func(*x) for x in entries],
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow/python/util/nest.py", line 914, in
structure[0], [func(*x) for x in entries],
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow_datasets/core/download/download_manager.py", line 767, in
res = tf.nest.map_structure(lambda p: p.get(), all_promises) # Wait promises
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/promise/promise.py", line 512, in get
return self._target_settled_value(_raise=True)
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/promise/promise.py", line 516, in _target_settled_value
return self._target()._settled_value(_raise)
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/promise/promise.py", line 226, in _settled_value
reraise(type(raise_val), raise_val, self._traceback)
File "/home/marstan/.local/lib/python3.8/site-packages/six.py", line 719, in reraise
raise value
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/promise/promise.py", line 844, in handle_future_result
resolve(future.result())
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow_datasets/core/download/downloader.py", line 216, in _sync_download
with _open_url(url, verify=verify) as (response, iter_content):
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/contextlib.py", line 113, in enter
return next(self.gen)
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow_datasets/core/download/downloader.py", line 278, in _open_with_requests
_assert_status(response)
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow_datasets/core/download/downloader.py", line 309, in _assert_status
raise DownloadError('Failed to get url {}. HTTP code: {}.'.format(
tensorflow_datasets.core.download.downloader.DownloadError: Failed to get url https://www.statmt.org/lm-benchmark/1-billion-word-language-modeling-benchmark-r13output.tar.gz. HTTP code: 404.

Expected behavior
Successfully download and prepare.

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in tensorflow_datasets/text/lm1b.py, especially _split_generators, and reproduce the failure with the provided download_and_prepare() snippet. Check how the downloader handles the unavailable URL and the pre-downloaded .gz file. Done means Lm1b can be downloaded from an available source or prepared from the local archive successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
data, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.