Key-aware batching function
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
I have a CombineFn for which add_input has very large overhead. I would like to batch the incoming elements into a large batch before each call to add_input to reduce this overhead. In other words, I would like to do something like:
`elements | GroupByKey() | BatchElements() | CombineValues(MyCombineFn())`
Unfortunately, BatchElements is not key-aware, and can't be used after a GroupByKey to batch elements per key. I'm working around this by doing the batching within CombineValues, which makes the CombineFn rather messy. It would be nice if there were a key-aware BatchElements transform which could be used in this context.
Imported from Jira [BEAM-3737](https://issues.apache.org/jira/browse/BEAM-3737). Original Jira may contain additional context.
Reported by: chuanyu.
Contributor guide
Research direction
Start by reading the existing BatchElements, GroupByKey, and CombineValues transforms and the CombineFn add_input contract. Determine how batching can preserve per-key grouping and define the expected behavior for the key-aware transform, including how it should be tested. Done means a usable key-aware batching transform that fits the shown pipeline and has coverage for batching elements per key.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100