apache / apache/beam

Ensure that the payload is consumed on Elements.Data and Elements.Timers when is_last = true

Open
#21,164 3 comments 0 reactions 0 assignees View on GitHub
bug go harness java P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

BeamFnDataInboundObserver drops the bytes instead of passing it forward on the last message: https://github.com/apache/beam/blob/cd4b7f3b1af4f51bdab1a0b1a98f94b5288c09ec/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java#L58

data_plane.py drops the timers payload but seems to do the right thing for data:
https://github.com/apache/beam/blob/cd4b7f3b1af4f51bdab1a0b1a98f94b5288c09ec/sdks/python/apache_beam/runners/worker/data_plane.py#L343

data_mgr.go drops the data payload (timers don't seem to be supported):
https://github.com/apache/beam/blob/cd4b7f3b1af4f51bdab1a0b1a98f94b5288c09ec/sdks/go/pkg/beam/core/runtime/harness/datamgr.go#L297

Imported from Jira [BEAM-13142](https://issues.apache.org/jira/browse/BEAM-13142). Original Jira may contain additional context.
Reported by: lcwik.

Contributor guide

Open the contributing guide

Research direction

Start with BeamFnDataInboundObserver.java, data_plane.py, and datamgr.go at the referenced locations, comparing how final Elements.Data and Elements.Timers messages are handled. Confirm that payload bytes are forwarded when is_last=true in each supported path, then verify the relevant runner harness behavior with existing tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, java, python
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.