apache / apache/beam

Consider: Go ViewFn support

Open
#18,602 0 comments 0 reactions 0 assignees View on GitHub
go improvement P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

We can add it as an optional field to beam.SideInput (see: pkg/beam/option.go). The execution side needs to support it as well. However, with the various side input forms, it's not clear how valuable such a feature would be.

It is strongly recommended to add Map Side Inputs https://issues.apache.org/jira/browse/BEAM-3293 before implementing this suggestion, and required to have caching implemented https://issues.apache.org/jira/browse/BEAM-11097. Otherwise very little benefit is acheived.

See https://issues.apache.org/jira/browse/BEAM-3293 for where code might need to be changed.

Make providing a ViewFn part of the beam.SideInput struct, either as a method to validate & update unexported fields, or adding an optional field. 

The ViewFn would be a function that takes in the iterator/multimap version of the Side Input and returns whatever the pre-processed type is. That value would be what is cached to reduce work per common window.

The DoFn should then be able to use that type in the position of the ProcessElement method for that side input. This avoids issues that only work in of the Global Window like trying to process the side input in StartBundle. Something would need to be changed about handling the DoFn method signature analysis to support additional alternative side input types.

Imported from Jira [BEAM-3305](https://issues.apache.org/jira/browse/BEAM-3305). Original Jira may contain additional context.
Reported by: herohde.

Contributor guide

Open the contributing guide

Research direction

Start with pkg/beam/option.go and the changes identified by BEAM-3293, then review the caching prerequisite in BEAM-11097. Trace how the execution side handles side-input forms and how DoFn method signatures are analyzed. Done means an optional ViewFn can be validated, applied and cached for supported side inputs without restricting processing to the Global Window.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.