seqio_cache_tasks fails on DataflowRunner
- Dominant language
- Python
- Stars
- 596
- Forks
- 60
- Avg merge
- 19h 23m
- Merged PRs (30d)
- 1
Description
When trying to cache a dataset that does not fit DirectRunner (e.g https://github.com/google-research/text-to-text-transfer-transformer/issues/323#issuecomment-667091989) on Cloud Dataflow without any `requirements.txt`, like
```
python -m seqio.scripts.cache_tasks_main \
--module_import="..." \
--tasks="${TASK_NAME}" \
--output_cache_dir="${BUCKET}/cache" \
--alsologtostderr \
--pipeline_options="--runner=DataflowRunner,--project=$PROJECT,--region=$REGION,--job_name=$TASK_NAME,--staging_location=$BUCKET/binaries,--temp_location=$BUCKET/tmp,--experiments=shuffle_mode=appliance"
```
it fails with `ModuleNotFoundError: No module named 'seqio'`.
If `seqio` added with
```
echo seqio > /tmp/beam_requirements.txt
# and run the same, adding to `--pipeline_options`
--requirements_file=/tmp/beam_requirements.txt
```
it fails with
```
subprocess.CalledProcessError: Command '['.../.venv/bin/python', '-m', 'pip', 'download', '--dest', '..../pip-tmp
/dataflow-requirements-cache', '-r', '/tmp/beam_requirements.txt', '--exists-action', 'i', '--no-binary', ':all:']' returned non-zero exit status 1.
Pip install failed for package: -r
Output from execution of subprocess: b"ERROR: Could not find a version that satisfies the requirement tensorflow-text (from versions: none)\
nERROR: No matching distribution found for tensorflow-text
```
This seems to be cause by `seqio` depending on `tensorflow-text`, which does not have any source release artifacts.
But [requirements cache in Apache Beam](https://github.com/apache/beam/blob/2aab1b04c2f5002527e0f2d25075b282feb7c054/sdks/python/apache_beam/runners/portability/stager.py#L641) seem to be populated with `--no-binary :all:` before making it available to the workers.
A try on a clean venv results in the same:
```
pip3 install --no-binary :all: --no-deps tensorflow-text==2.6.0
ERROR: Could not find a version that satisfies the requirement tensorflow-text==2.6.0 (from versions: none)
ERROR: No matching distribution found for tensorflow-text==2.6.0
```
Am I doing something wrong, or how does everyone work this around? Would appreciate a hand here.
Contributor guide
Assessment
This issue has not been assessed yet.