apache / apache/beam

DoFn per-key lifecycle (Setup/Teardown)

Open
#18,150 0 comments 0 reactions 0 assignees View on GitHub
beam-model P3 wish
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

DoFn's that use state and timers are implicitly per-key. Setup/Teardown methods are usually used to establish expensive resources - long-standing connections and such.

For per-key DoFn's, we'd often want to use these per-key, so it'd be good to have ability in the model to ask that there be 1 instance of the DoFn per key, reused between e.g. different timer or trigger firings for this key, but not used for other keys.

E.g. this would be particularly useful for Splittable DoFn - for its ability to reuse expensive resources between checkpoints.

Imported from Jira [BEAM-1314](https://issues.apache.org/jira/browse/BEAM-1314). Original Jira may contain additional context.
Reported by: jkff.

Contributor guide

Open the contributing guide

Research direction

Start with the original Jira issue BEAM-1314 and the DoFn lifecycle model described here, focusing on Setup/Teardown, state and timers, and Splittable DoFn checkpoints. Done would mean a defined model for per-key DoFn instances and their reuse across timer or trigger firings without sharing them between keys.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.