apache / apache/beam

Key-aware batching function

Open
#18,634 0 comments 0 reactions 0 assignees View on GitHub
core new feature P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

I have a CombineFn for which add_input has very large overhead. I would like to batch the incoming elements into a large batch before each call to add_input to reduce this overhead. In other words, I would like to do something like: 

`elements | GroupByKey() | BatchElements() | CombineValues(MyCombineFn())`

Unfortunately, BatchElements is not key-aware, and can't be used after a GroupByKey to batch elements per key. I'm working around this by doing the batching within CombineValues, which makes the CombineFn rather messy. It would be nice if there were a key-aware BatchElements transform which could be used in this context.

Imported from Jira [BEAM-3737](https://issues.apache.org/jira/browse/BEAM-3737). Original Jira may contain additional context.
Reported by: chuanyu.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing BatchElements, GroupByKey, and CombineValues transforms and the CombineFn add_input contract. Determine how batching can preserve per-key grouping and define the expected behavior for the key-aware transform, including how it should be tested. Done means a usable key-aware batching transform that fits the shown pipeline and has coverage for batching elements per key.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.