apache / apache/beam

WindmillStateCache grossly misunderestimates object size in its weighting function

Open
#19,645 0 comments 0 reactions 0 assignees View on GitHub
bug dataflow P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

As a little experiment, we set the WindmillStateCache size to 2 GB.  After we let a job run for a few hours, the workers all died from OOMs.  Digging into the memory usage, over 10 GB of heap was used by the WindmillStateCache.  The total weight the cache had calculated however was only ~800 MB.

It looks like the StateCacheEntry calculation is way off.  The weight of an entry was calculated as only ~200, while the actual heap usage of it (as reported by YourKit at least) was around 2,400 bytes.

Imported from Jira [BEAM-7684](https://issues.apache.org/jira/browse/BEAM-7684). Original Jira may contain additional context.
Reported by: SteveNiemitz.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.