apache / apache/beam

Verbose print causes pipelines to get stuck on FlinkRunner.

Open
#21,589 0 comments 0 reactions 0 assignees View on GitHub
bug flink harness P2 python runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Minimal reproduction: 
 
`def _run_pipeline(pipeline):`
`    def process_data(unused):`
`        print('a'*1000)`

`    _ = (`
`            pipeline`
`            | "Create" >> beam.Create(['a']*1000)`
`            | "Process" >> beam.Map(process_data)`
`    )`
`    pipeline.run().wait_until_finish()`

See thread at https://lists.apache.org/thread/r971f432mvko4g3hmnrodh8s4n2wkk6p

Imported from Jira [BEAM-14476](https://issues.apache.org/jira/browse/BEAM-14476). Original Jira may contain additional context.
Reported by: robertwb.

Contributor guide

Open the contributing guide

Research direction

Start with the minimal _run_pipeline reproduction and the linked Apache mailing-list thread, using FlinkRunner and pipeline.run().wait_until_finish(). Trace why repeated print output causes the pipeline to stop progressing; done means the reproduction completes without getting stuck and the expected output is handled correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
distributed-systems, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.