apache / apache/beam

[Go SDK] Add support for setting upper memory limit for side input cache

Open
#21,129 0 comments 0 reactions 0 assignees View on GitHub
go improvement P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

This is implementing https://issues.apache.org/jira/browse/BEAM-8297 for the Go SDK's side input cache. The current implementation reads the entire side input into memory and keeps track of how many distinct inputs to hold; however, each side input is not guaranteed to occupy the same amount of memory. An extra parameter should be created to be passed in to the SideInputCache ([https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/harness/statecache/statecache.go)](https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/harness/statecache/statecache.go)) to set a maximum memory limit in place of a maximum element capacity, then add a hook (see https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/harness/cache_hooks.go) and corresponding function to the harnessopts package ([https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/util/harnessopts/cache.go)](https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/util/harnessopts/cache.go)) so users can set this value themselves. 

Imported from Jira [BEAM-13212](https://issues.apache.org/jira/browse/BEAM-13212). Original Jira may contain additional context.
Reported by: jrmccluskey.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.