Portable Wordcount on Flink runner finishes but does not produce output files in DOCKER execution mode
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 204
Description
Repro steps:
```
./gradlew :runners:flink:1.5:job-server:runShadow
./gradlew :sdks:python:container:py3:docker
rm
-rf /tmp/py-wordcount-direct*
./gradlew :sdks:python:test-suites:portable:py35:portableWordCountBatch
-PjobEndpoint=localhost:8099 -PenvironmentType=DOCKER
cat /tmp/py-wordcount-direct* # to verify results.
#
Output: No such file or directory
```
Switching to -PenvironmentType=LOOPBACK produces output.
The issue is reproducible both on Py2 and Py3 on master.
cc: [~altay] [~angoenka] [~ibzib] [~robertwb]
Imported from Jira [BEAM-7857](https://issues.apache.org/jira/browse/BEAM-7857). Original Jira may contain additional context.
Reported by: tvalentyn.
Contributor guide
Research direction
Start with the runners/flink/1.5/job-server:runShadow entry point and the sdks/python/test-suites/portable/py35/portableWordCountBatch task. Reproduce in DOCKER mode and compare it with LOOPBACK, checking why the expected /tmp/py-wordcount-direct* output is absent. Done means the portable Wordcount job produces its output files in DOCKER execution for both Py2 and Py3.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, java, python
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100