apache / apache/beam

Add SetUp() and TearDown() for CombineFns

Open
#18,716 0 comments 0 reactions 0 assignees View on GitHub
beam-model core improvement P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

I have a CombineFn that has a large amount of state that needs to be loaded once before it can add_input or merge_combiners (for example, the CombineFn might load up a large lookup table used for combining). 

Right now, to initialise this state, for each of the methods, I check if the state has already been initialised, and if not, I initialise it. It would be nice if CombineFn provided a SetUp() method that is called once to initialise this state (and a corresponding TearDown() method to clean up this state if necessary).

Imported from Jira [BEAM-3736](https://issues.apache.org/jira/browse/BEAM-3736). Original Jira may contain additional context.
Reported by: chuanyu.

Contributor guide

Open the contributing guide

Research direction

Start by locating CombineFn and its add_input and merge_combiners methods. Review how state is initialized and cleaned up today, then determine the lifecycle and runner semantics needed for SetUp() and TearDown(); done means the requested hooks are specified and covered by appropriate tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.