apache / apache/beam

Portable Wordcount on Flink runner finishes but does not produce output files in DOCKER execution mode

Open
#19,726 0 comments 0 reactions 0 assignees View on GitHub
bug core flink P3 portability python runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 5h
Merged PRs (30d)
204

Description

Repro steps:

```

./gradlew :runners:flink:1.5:job-server:runShadow   
./gradlew :sdks:python:container:py3:docker
rm
-rf /tmp/py-wordcount-direct*
./gradlew :sdks:python:test-suites:portable:py35:portableWordCountBatch
 -PjobEndpoint=localhost:8099 -PenvironmentType=DOCKER
cat /tmp/py-wordcount-direct* # to verify results.

#
Output: No such file or directory

```

Switching to -PenvironmentType=LOOPBACK produces output.
The issue is reproducible both on Py2 and Py3 on master.

cc: [~altay] [~angoenka] [~ibzib] [~robertwb]

Imported from Jira [BEAM-7857](https://issues.apache.org/jira/browse/BEAM-7857). Original Jira may contain additional context.
Reported by: tvalentyn.

Contributor guide

Open the contributing guide

Research direction

Start with the runners/flink/1.5/job-server:runShadow entry point and the sdks/python/test-suites/portable/py35/portableWordCountBatch task. Reproduce in DOCKER mode and compare it with LOOPBACK, checking why the expected /tmp/py-wordcount-direct* output is absent. Done means the portable Wordcount job produces its output files in DOCKER execution for both Py2 and Py3.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java, python
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.