tensorflow / tensorflow/datasets
Cannot download_and_prepare Lm1b dataset
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
/!\ PLEASE INCLUDE THE FULL STACKTRACE AND CODE SNIPPET
Short description
The host website of Lm1b dataset is down, so downloading Lm1b dataset returns a 404 error. I have managed to download the .gz file of Lm1b dataset, however I found it hard to prepare Lm1b with a pre-downloaded gz file since I cannot call prepare() separately. Is there a way to find a Lm1b dataset mirror or to load Lm1b dataset from .gz file?
Environment information
-
Operating System: Ubuntu 18.04
-
Python version: 3.8.13
-
tensorflow-datasets/tfds-nightlytensorflow-datasets 4.4.0 -
tensorflow/tf-nightlyversion: tensorflow 2.8.1 -
Does the issue still exists with the last
tfds-nightlypackage (pip install --upgrade tfds-nightly) ?
yes
Reproduction instructions
import tensorflow_datasets as tfds
builder = tfds.text.Lm1b()
builder.download_and_prepare()
If you share a colab, make sure to update the permissions to share it.
Link to logs
Downloading and preparing dataset 1.67 GiB (download: 1.67 GiB, generated: 4.40 GiB, total: 6.07 GiB) to /home/marstan/tensorflow_datasets/lm1b/1.1.0...
Extraction completed...: 0 file [00:01, ? file/s] | 0/1 [00:00<?, ? url/s]
Dl Size...: 0 MiB [00:01, ? MiB/s]
Dl Completed...: 0%| | 0/1 [00:01<?, ? url/s]
Traceback (most recent call last):
File "", line 1, in
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow_datasets/core/dataset_builder.py", line 439, in download_and_prepare
self._download_and_prepare(
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow_datasets/core/dataset_builder.py", line 1113, in _download_and_prepare
split_generators = self._split_generators( # pylint: disable=unexpected-keyword-arg
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow_datasets/text/lm1b.py", line 87, in _split_generators
lm1b_path = dl_manager.download_and_extract(_DOWNLOAD_URL)
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow_datasets/core/download/download_manager.py", line 634, in download_and_extract
return _map_promise(self._download_extract, url_or_urls)
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow_datasets/core/download/download_manager.py", line 767, in _map_promise
res = tf.nest.map_structure(lambda p: p.get(), all_promises) # Wait promises
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow/python/util/nest.py", line 914, in map_structure
structure[0], [func(*x) for x in entries],
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow/python/util/nest.py", line 914, in
structure[0], [func(*x) for x in entries],
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow_datasets/core/download/download_manager.py", line 767, in
res = tf.nest.map_structure(lambda p: p.get(), all_promises) # Wait promises
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/promise/promise.py", line 512, in get
return self._target_settled_value(_raise=True)
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/promise/promise.py", line 516, in _target_settled_value
return self._target()._settled_value(_raise)
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/promise/promise.py", line 226, in _settled_value
reraise(type(raise_val), raise_val, self._traceback)
File "/home/marstan/.local/lib/python3.8/site-packages/six.py", line 719, in reraise
raise value
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/promise/promise.py", line 844, in handle_future_result
resolve(future.result())
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow_datasets/core/download/downloader.py", line 216, in _sync_download
with _open_url(url, verify=verify) as (response, iter_content):
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/contextlib.py", line 113, in enter
return next(self.gen)
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow_datasets/core/download/downloader.py", line 278, in _open_with_requests
_assert_status(response)
File "/home/marstan/anaconda3/envs/flax_test/lib/python3.8/site-packages/tensorflow_datasets/core/download/downloader.py", line 309, in _assert_status
raise DownloadError('Failed to get url {}. HTTP code: {}.'.format(
tensorflow_datasets.core.download.downloader.DownloadError: Failed to get url https://www.statmt.org/lm-benchmark/1-billion-word-language-modeling-benchmark-r13output.tar.gz. HTTP code: 404.
Expected behavior
Successfully download and prepare.
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in tensorflow_datasets/text/lm1b.py, especially _split_generators, and reproduce the failure with the provided download_and_prepare() snippet. Check how the downloader handles the unavailable URL and the pre-downloaded .gz file. Done means Lm1b can be downloaded from an available source or prepared from the local archive successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100