[Feature Request]: Bound PGBKCVOperation precombine table by in-memory size, not key count
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What would you like to happen?
`PGBKCVOperation` (`sdks/python/apache_beam/runners/worker/operations.py`) caps its precombine table by key count (`max_keys`, default 100k / 1M for Count/Mean/min/max/sum), flushing ~10% of keys once the count is hit. The code already flags this as a stopgap:
```python
# TODO(b/36567833): Bound by in-memory size rather than key count.
```
A fixed key-count cap is a poor memory proxy when accumulators vary in size, so workers can OOM well below `max_keys` (or flush needlessly when accumulators are tiny). Request: bound the table by estimated in-memory size instead of (or in addition to) key count.
Notes:
- `b/36567833` is a Google-internal Buganizer ref with no public page; this issue tracks it publicly.
- Sibling `PGBKOperation` has the same shape (`max_size = 10 * 1000` elements).
- Sizing accurately is the hard part — `sys.getsizeof` undercounts wrapped/native (e.g. C-extension) accumulators; a coder-based or sampled-RSS estimate is likely needed.
### Issue Priority
Priority: 3 (nice-to-have improvement)
### Issue Components
- Component: Python SDK
Contributor guide
Research direction
Start in sdks/python/apache_beam/runners/worker/operations.py at PGBKCVOperation and its TODO about bounding by in-memory size. Compare the sibling PGBKOperation and inspect how the Count, Mean, min, max, and sum accumulators are represented. Done means the precombine table is bounded using an in-memory-size estimate rather than relying only on key count, while accounting for wrapped or native accumulators.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100