apache / apache/beam

[Bug]: No way to read or write to file when running Beam in Flink

Open
#21,946 3 comments 0 reactions 0 assignees View on GitHub
flink runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

My code is (this is taken from Beam examples):

```
def run():
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions

options = PipelineOptions([
"--runner=FlinkRunner",
"--flink_version=1.14",
"--flink_master=localhost:8081",
"--environment_config=localhost:50000"
])

output_file_prefix = 'C:\\ApacheBeam\\output'

with beam.Pipeline(options=options) as p:
(p
| 'Create file lines' >> beam.Create([
'Each element must be a string.',
'It writes one element per line.',
'There are no guarantees on the line order.',
'The data might be written into multiple files.',
])
| 'Write to files' >> beam.io.WriteToText(output_file_prefix, file_name_suffix='.txt')
)

if __name__ == "__main__":
run()
```

But Flink is not able to write or read from file:

`Caused by: java.lang.Exception: The user defined 'open()' method caused an exception: java.io.IOException: Cannot run program "docker": CreateProcess error=2, The system cannot find the file specified`

According Beam documentation this is how it should work: https://beam.apache.org/documentation/runners/flink/

### Issue Priority

Priority: 1

### Issue Component

Component: runner-flink

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.