DoFn per-key lifecycle (Setup/Teardown)
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
DoFn's that use state and timers are implicitly per-key. Setup/Teardown methods are usually used to establish expensive resources - long-standing connections and such.
For per-key DoFn's, we'd often want to use these per-key, so it'd be good to have ability in the model to ask that there be 1 instance of the DoFn per key, reused between e.g. different timer or trigger firings for this key, but not used for other keys.
E.g. this would be particularly useful for Splittable DoFn - for its ability to reuse expensive resources between checkpoints.
Imported from Jira [BEAM-1314](https://issues.apache.org/jira/browse/BEAM-1314). Original Jira may contain additional context.
Reported by: jkff.
Contributor guide
Research direction
Start with the original Jira issue BEAM-1314 and the DoFn lifecycle model described here, focusing on Setup/Teardown, state and timers, and Splittable DoFn checkpoints. Done would mean a defined model for per-key DoFn instances and their reuse across timer or trigger firings without sharing them between keys.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100