tensorflow / tensorflow/datasets
`wmt_t2t_translate` fails build when `--data_dir` set to GCS location
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
/!\ PLEASE INCLUDE THE FULL STACKTRACE AND CODE SNIPPET
Short description
wmt_t2t_translate will not build when --data_dir is set to a GCS location due to etils.Path().glob() behavior when glob pattern is a gcs URI
Environment information
-
Operating System: Linux
-
Python version: 3.9.15
-
tensorflow-datasets/tfds-nightlyversion: 4.8.2 / 4.8.2.dev202301270045 -
tensorflow/tf-nightlyversion: 2.11.0 -
Does the issue still exists with the last
tfds-nightlypackage (pip install --upgrade tfds-nightly) ? -
yes
Reproduction instructions
pip install tensorflow tensorflow-datasetstfds build --data_dir $GCS_DATA_DIR wmt_t2t_translate
INFO[wmt.py]: Using sub-datasets: {Split('train'): ['europarl_v7', 'commoncrawl', 'newscommentary_v13'], Split('validation'): ['newstest2013'], Split('test'): ['newstest2014']}
INFO[wmt.py]: Generating examples from: europarl_v7
Traceback (most recent call last):
File "/opt/conda/envs/tfds-bug/bin/tfds", line 8, in <module>
sys.exit(launch_cli())
File "/opt/conda/envs/tfds-bug/lib/python3.9/site-packages/tensorflow_datasets/scripts/cli/main.py", line 109, in launch_cli
app.run(main, flags_parser=_parse_flags)
File "/opt/conda/envs/tfds-bug/lib/python3.9/site-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/opt/conda/envs/tfds-bug/lib/python3.9/site-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
File "/opt/conda/envs/tfds-bug/lib/python3.9/site-packages/tensorflow_datasets/scripts/cli/main.py", line 104, in main
args.subparser_fn(args)
File "/opt/conda/envs/tfds-bug/lib/python3.9/site-packages/tensorflow_datasets/scripts/cli/build.py", line 274, in _build_datasets
_download_and_prepare(args, builder)
File "/opt/conda/envs/tfds-bug/lib/python3.9/site-packages/tensorflow_datasets/scripts/cli/build.py", line 510, in _download_and_prepare
builder.download_and_prepare(
File "/opt/conda/envs/tfds-bug/lib/python3.9/site-packages/tensorflow_datasets/core/logging/__init__.py", line 169, in __call__
return function(*args, **kwargs)
File "/opt/conda/envs/tfds-bug/lib/python3.9/site-packages/tensorflow_datasets/core/dataset_builder.py", line 642, in download_and_prepare
self._download_and_prepare(
File "/opt/conda/envs/tfds-bug/lib/python3.9/site-packages/tensorflow_datasets/core/dataset_builder.py", line 1487, in _download_and_prepare
future = split_builder.submit_split_generation(
File "/opt/conda/envs/tfds-bug/lib/python3.9/site-packages/tensorflow_datasets/core/split_builder.py", line 340, in submit_split_generation
return self._build_from_generator(**build_kwargs)
File "/opt/conda/envs/tfds-bug/lib/python3.9/site-packages/tensorflow_datasets/core/split_builder.py", line 406, in _build_from_generator
for key, example in utils.tqdm(
File "/opt/conda/envs/tfds-bug/lib/python3.9/site-packages/tqdm/std.py", line 1195, in __iter__
for obj in iterable:
File "/opt/conda/envs/tfds-bug/lib/python3.9/site-packages/tensorflow_datasets/translate/wmt.py", line 938, in _generate_examples
for sub_key, ex in sub_generator(*files):
File "/opt/conda/envs/tfds-bug/lib/python3.9/site-packages/tensorflow_datasets/translate/wmt.py", line 1000, in _parse_parallel_sentences
assert f1_files and f2_files, "No matching files found: %s, %s." % (f1, f2)
AssertionError: No matching files found: gs://gcp-ml-sandbox-t5x/datasets/downloads/extracted/TAR_GZ.statmt.org_wmt13_traini-parall-europa-v7AiTHxxDIoGPf2JOwzAgwIC1h9MdcF-uOMYNhA9J9luc.tgz/training/europarl-v7.de-en.de, gs://gcp-ml-sandbox-t5x/datasets/downloads/extracted/TAR_GZ.statmt.org_wmt13_traini-parall-europa-v7AiTHxxDIoGPf2JOwzAgwIC1h9MdcF-uOMYNhA9J9luc.tgz/training/europarl-v7.de-en.en.
(tfds-bug) jupyter@python3-conda-gpu:~/T5X$
If you share a colab, make sure to update the permissions to share it.
Link to logs
If applicable, <link to gist with logs, stack trace>
Expected behavior
dataset to be built
Additional context
Seems to be an issue with file globbing behavior being different for gcs vs local paths
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the tfds build --data_dir $GCS_DATA_DIR wmt_t2t_translate reproduction and read tensorflow_datasets/translate/wmt.py, especially _parse_parallel_sentences near the failing assertion. Compare how etils.Path().glob() handles the GCS paths shown in the traceback versus local paths. Done means the dataset builds successfully with a GCS --data_dir without the no-matching-files assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100