Instructions for running Beam Python on Flink do not work on Mac
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
I am trying to run Beam on a separate Flink cluster, as described here: [https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster](https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster)
??(1) Start a Flink cluster which exposes the Rest interface (e.g. `localhost:8081` by default).??
Flink 1.13.1 starts successfully on localhost:8081.
??(2) Start JobService with Flink Rest endpoint: `docker run --net=host apache/beam_flink1.10_job_server:latest --flink-master=localhost:8081`.??
I am running this instead, which again starts successfully:
`docker run --net=host apache/beam_flink1.13_job_server:latest --flink-master=localhost:8081`
??(3) Submit the pipeline as above.??
I'm running the WordCount pipeline as:
`python -m apache_beam.examples.wordcount --input /Users/stefan/datastore/input.txt`
` --output /Users/stefan/datastore/output.txt`
` --runner=PortableRunner --job_endpoint=localhost:8099`
After 1 minute the pipeline crashes with a `grpc.FutureTimeoutError`. This is the full output:
`WARNING:root:Make sure that locally built Python SDK docker image has Python 3.8 interpreter.`
`INFO:root:Default Python SDK image for environment is apache/beam_python3.8_sdk:2.31.0`
`INFO:root:No image given, using default Python SDK image`
`WARNING:root:Make sure that locally built Python SDK docker image has Python 3.8 interpreter.`
`INFO:root:Default Python SDK image for environment is apache/beam_python3.8_sdk:2.31.0`
`INFO:root:Python SDK container image set to "apache/beam_python3.8_sdk:2.31.0" for Docker environment`
`INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== ====================`
`INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== ====================`
`INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== ====================`
`Traceback (most recent call last):`
\` File "/usr/local/Cellar/python@3.8/3.8.11/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 194, in _run_module_as_main`
\` return _run_code(code, main_globals, None,`
\` File "/usr/local/Cellar/python@3.8/3.8.11/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code`
\` exec(code, run_globals)`
\` File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/examples/wordcount.py", line 94, in `
\` run()`
\` File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/examples/wordcount.py", line 89, in run`
\` output | 'Write' >> WriteToText(known_args.output)`
` File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/pipeline.py", line 585, in __exit__`
\` self.result = self.run()`
\` File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/pipeline.py", line 564, in run`
\` return self.runner.run_pipeline(self, self._options)`
\` File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/runners/portability/portable_runner.py", line 438, in run_pipeline`
\` job_service_handle = self.create_job_service(options)`
\` File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/runners/portability/portable_runner.py", line 317, in create_job_service`
\` return self.create_job_service_handle(server.start(), options)`
\` File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/runners/portability/job_server.py", line 54, in start`
\` grpc.channel_ready_future(channel).result(timeout=self._timeout)`
\` File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/grpc/_utilities.py", line 140, in result`
\` self._block(timeout)`
\` File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/grpc/_utilities.py", line 86, in _block`
\` raise grpc.FutureTimeoutError()`
`grpc.FutureTimeoutError`
What's going on? Any help would be appreciated.
Imported from Jira [BEAM-12657](https://issues.apache.org/jira/browse/BEAM-12657). Original Jira may contain additional context.
Reported by: stefanistrate.
Contributor guide
Assessment
This issue has not been assessed yet.