apache / apache/beam

Go SDK-Tune in memory pre-combine caching for Lifted Combines.

Open
#18,944 0 comments 0 reactions 0 assignees View on GitHub
go improvement P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Requires [BEAM-4276](https://issues.apache.org/jira/browse/BEAM-4276) to be completed first.

Additional performance tweaks to the in memory per-key accumulator cache during the pre-combine phase of a lifted combine.

This can include any of
* capping the number of key-accumulator in the cache, and draining them eagerly after x elements seen, or evicting keys by some heuristic after the cache has seen X distinct keys.
* providing a counter on cache size (key/element counts) exposable through the metrics interface or another runner standard counter, to permit observing the cache's status, especially if it could grow without bound within a bundle.

 

Imported from Jira [BEAM-4468](https://issues.apache.org/jira/browse/BEAM-4468). Original Jira may contain additional context.
Reported by: lostluck.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.