Consider: Go ViewFn support
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
We can add it as an optional field to beam.SideInput (see: pkg/beam/option.go). The execution side needs to support it as well. However, with the various side input forms, it's not clear how valuable such a feature would be.
It is strongly recommended to add Map Side Inputs https://issues.apache.org/jira/browse/BEAM-3293 before implementing this suggestion, and required to have caching implemented https://issues.apache.org/jira/browse/BEAM-11097. Otherwise very little benefit is acheived.
See https://issues.apache.org/jira/browse/BEAM-3293 for where code might need to be changed.
Make providing a ViewFn part of the beam.SideInput struct, either as a method to validate & update unexported fields, or adding an optional field.
The ViewFn would be a function that takes in the iterator/multimap version of the Side Input and returns whatever the pre-processed type is. That value would be what is cached to reduce work per common window.
The DoFn should then be able to use that type in the position of the ProcessElement method for that side input. This avoids issues that only work in of the Global Window like trying to process the side input in StartBundle. Something would need to be changed about handling the DoFn method signature analysis to support additional alternative side input types.
Imported from Jira [BEAM-3305](https://issues.apache.org/jira/browse/BEAM-3305). Original Jira may contain additional context.
Reported by: herohde.
Contributor guide
Research direction
Start with pkg/beam/option.go and the changes identified by BEAM-3293, then review the caching prerequisite in BEAM-11097. Trace how the execution side handles side-input forms and how DoFn method signatures are analyzed. Done means an optional ViewFn can be validated, applied and cached for supported side inputs without restricting processing to the Global Window.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100