[Bug]: No way to read or write to file when running Beam in Flink
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
My code is (this is taken from Beam examples):
```
def run():
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
options = PipelineOptions([
"--runner=FlinkRunner",
"--flink_version=1.14",
"--flink_master=localhost:8081",
"--environment_config=localhost:50000"
])
output_file_prefix = 'C:\\ApacheBeam\\output'
with beam.Pipeline(options=options) as p:
(p
| 'Create file lines' >> beam.Create([
'Each element must be a string.',
'It writes one element per line.',
'There are no guarantees on the line order.',
'The data might be written into multiple files.',
])
| 'Write to files' >> beam.io.WriteToText(output_file_prefix, file_name_suffix='.txt')
)
if __name__ == "__main__":
run()
```
But Flink is not able to write or read from file:
`Caused by: java.lang.Exception: The user defined 'open()' method caused an exception: java.io.IOException: Cannot run program "docker": CreateProcess error=2, The system cannot find the file specified`
According Beam documentation this is how it should work: https://beam.apache.org/documentation/runners/flink/
### Issue Priority
Priority: 1
### Issue Component
Component: runner-flink
Contributor guide
Assessment
This issue has not been assessed yet.