apache / apache/beam

[Feature Request]: [Go SDK] Encode Keys once for combiner lifting.

Open
#22,704 1 comment 0 reactions 0 assignees View on GitHub
go new feature P2 performance
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

The Go SDK's combiner lifting currently encodes keys twice for caching. First it encodes values into a hash function, and then it repeats it to get the []byte for byte by byte comparisons in the cache.

https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/combine.go#L469

https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/combine.go#L483

For expensive keys, this can become very expensive without additional caching schemes.

Ideally this encoding happens once, instead of twice for each key being looked up.

Note: Windowed byte equality is important to use for final lookups because that's the only way to ensure equivalence to runners. Beam Model GBKs use byte equality for keys as their basis.

### Issue Priority

Priority: 2

### Issue Component

Component: sdk-go

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.