[Go SDK] Add support for setting upper memory limit for side input cache
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
This is implementing https://issues.apache.org/jira/browse/BEAM-8297 for the Go SDK's side input cache. The current implementation reads the entire side input into memory and keeps track of how many distinct inputs to hold; however, each side input is not guaranteed to occupy the same amount of memory. An extra parameter should be created to be passed in to the SideInputCache ([https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/harness/statecache/statecache.go)](https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/harness/statecache/statecache.go)) to set a maximum memory limit in place of a maximum element capacity, then add a hook (see https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/harness/cache_hooks.go) and corresponding function to the harnessopts package ([https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/util/harnessopts/cache.go)](https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/util/harnessopts/cache.go)) so users can set this value themselves.
Imported from Jira [BEAM-13212](https://issues.apache.org/jira/browse/BEAM-13212). Original Jira may contain additional context.
Reported by: jrmccluskey.
Contributor guide
Assessment
This issue has not been assessed yet.