apache / apache/beam

Enable caching of side-input dependent computations

Open
#18,090 0 comments 0 reactions 0 assignees View on GitHub
beam-model new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Sometimes the kind of computations one wants to perform in startBundle depend on side inputs (and, implicitly, the window). For example, one might want to initialize a (non-serializable) stateful object. In particular, this leads to users incorrectly (in the case of triggered or non-globally-windowed side inputs) memoizing this computation in the first processElement call.

One option would be to fold this into a customizable ViewFn.

Imported from Jira [BEAM-1002](https://issues.apache.org/jira/browse/BEAM-1002). Original Jira may contain additional context.
Reported by: robertwb.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how startBundle receives side inputs and how the customizable ViewFn option fits the existing Beam model. Define caching semantics for triggered and non-globally-windowed side inputs, including when window-dependent state must be refreshed; the work is done when those computations are cached correctly without first-element memoization.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.