Ensure that the payload is consumed on Elements.Data and Elements.Timers when is_last = true
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
BeamFnDataInboundObserver drops the bytes instead of passing it forward on the last message: https://github.com/apache/beam/blob/cd4b7f3b1af4f51bdab1a0b1a98f94b5288c09ec/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java#L58
data_plane.py drops the timers payload but seems to do the right thing for data:
https://github.com/apache/beam/blob/cd4b7f3b1af4f51bdab1a0b1a98f94b5288c09ec/sdks/python/apache_beam/runners/worker/data_plane.py#L343
data_mgr.go drops the data payload (timers don't seem to be supported):
https://github.com/apache/beam/blob/cd4b7f3b1af4f51bdab1a0b1a98f94b5288c09ec/sdks/go/pkg/beam/core/runtime/harness/datamgr.go#L297
Imported from Jira [BEAM-13142](https://issues.apache.org/jira/browse/BEAM-13142). Original Jira may contain additional context.
Reported by: lcwik.
Contributor guide
Research direction
Start with BeamFnDataInboundObserver.java, data_plane.py, and datamgr.go at the referenced locations, comparing how final Elements.Data and Elements.Timers messages are handled. Confirm that payload bytes are forwarded when is_last=true in each supported path, then verify the relevant runner harness behavior with existing tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, java, python
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100