google / google/seqio

seqio_cache_tasks fails on DataflowRunner

Open
#109 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
596
Forks
60
Avg merge
19h 23m
Merged PRs (30d)
1

Description

When trying to cache a dataset that does not fit DirectRunner (e.g https://github.com/google-research/text-to-text-transfer-transformer/issues/323#issuecomment-667091989) on Cloud Dataflow without any `requirements.txt`, like

```
python -m seqio.scripts.cache_tasks_main \
--module_import="..." \
--tasks="${TASK_NAME}" \
--output_cache_dir="${BUCKET}/cache" \
--alsologtostderr \
--pipeline_options="--runner=DataflowRunner,--project=$PROJECT,--region=$REGION,--job_name=$TASK_NAME,--staging_location=$BUCKET/binaries,--temp_location=$BUCKET/tmp,--experiments=shuffle_mode=appliance"
```

it fails with `ModuleNotFoundError: No module named 'seqio'`.

If `seqio` added with

```
echo seqio > /tmp/beam_requirements.txt

# and run the same, adding to `--pipeline_options`
--requirements_file=/tmp/beam_requirements.txt
```

it fails with
```
subprocess.CalledProcessError: Command '['.../.venv/bin/python', '-m', 'pip', 'download', '--dest', '..../pip-tmp
/dataflow-requirements-cache', '-r', '/tmp/beam_requirements.txt', '--exists-action', 'i', '--no-binary', ':all:']' returned non-zero exit status 1.

Pip install failed for package: -r
Output from execution of subprocess: b"ERROR: Could not find a version that satisfies the requirement tensorflow-text (from versions: none)\
nERROR: No matching distribution found for tensorflow-text
```

This seems to be cause by `seqio` depending on `tensorflow-text`, which does not have any source release artifacts.

But [requirements cache in Apache Beam](https://github.com/apache/beam/blob/2aab1b04c2f5002527e0f2d25075b282feb7c054/sdks/python/apache_beam/runners/portability/stager.py#L641) seem to be populated with `--no-binary :all:` before making it available to the workers.

A try on a clean venv results in the same:
```
pip3 install --no-binary :all: --no-deps tensorflow-text==2.6.0
ERROR: Could not find a version that satisfies the requirement tensorflow-text==2.6.0 (from versions: none)
ERROR: No matching distribution found for tensorflow-text==2.6.0
```

Am I doing something wrong, or how does everyone work this around? Would appreciate a hand here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.