apache / apache/beam

Flink's job server container attempts to invoke docker

Open
#21,085 0 comments 0 reactions 0 assignees View on GitHub
bug flink P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I want to run a Java pipeline with `--runner PortableRunner`, so I do the following:
* Run the Flink's job server with:

```

docker run --rm --net=host apache/beam_flink1.13_job_server:latest

```

* Run the pipeline with:

```

mvn exec:java -Dexec.mainClass=org.foo.bar.MyMainClass -Pportable-runner -Dexec.args="--runner=PortableRunner
--jobEndpoint=localhost:8099"

```

After this I get the error:
```

java.lang.Exception: The user defined 'open()' method caused an exception: java.io.IOException: Cannot
run program "docker": error=2, No such file or directory
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:513)
at
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:360)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
at java.lang.Thread.run(Thread.java:748)
```

What I did to workaround this is to use the `gradlew` script from the Git repository like this:
```

./gradlew :runners:flink:1.13:job-server:runShadow

```

and it works without problem.

Imported from Jira [BEAM-12642](https://issues.apache.org/jira/browse/BEAM-12642). Original Jira may contain additional context.
Reported by: itaranto.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Docker image command and the Java PortableRunner invocation shown in the report, then compare them with :runners:flink:1.13:job-server:runShadow. Trace why the container attempts to invoke docker and verify the job server can run the pipeline without requiring a docker executable.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.