apache / apache/beam

Bound the amount of data on the data plane by time.

Open
#19,236 0 comments 0 reactions 0 assignees View on GitHub
dataflow harness improvement java P3 python runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

This is especially important for Fn API reads, where each element represents a shard to read and may be very expensive, but many elements may be waiting in the Fn API buffer.

The need for this will be mitigated with full SDF support for liquid sharding over the Fn API, but not eliminated unless the runner can "unread" elements it has already sent. 

This is especially important in for dataflow jobs that start out small but then detect that they need more workers (e.g. due to the initial inputs being an SDF).

Imported from Jira [BEAM-5791](https://issues.apache.org/jira/browse/BEAM-5791). Original Jira may contain additional context.
Reported by: robertwb.

Contributor guide

Open the contributing guide

Research direction

Start by tracing Fn API reads, the data-plane buffer, and how runners send elements to workers. Clarify the intended time bound and how completion should be verified, since the issue does not name files, tests, or a precise acceptance condition.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
24/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.