apache / apache/beam

[Feature Request]: Bound PGBKCVOperation precombine table by in-memory size, not key count

Open
#39,754 0 comments 0 reactions 0 assignees View on GitHub
P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

`PGBKCVOperation` (`sdks/python/apache_beam/runners/worker/operations.py`) caps its precombine table by key count (`max_keys`, default 100k / 1M for Count/Mean/min/max/sum), flushing ~10% of keys once the count is hit. The code already flags this as a stopgap:

```python
# TODO(b/36567833): Bound by in-memory size rather than key count.
```

A fixed key-count cap is a poor memory proxy when accumulators vary in size, so workers can OOM well below `max_keys` (or flush needlessly when accumulators are tiny). Request: bound the table by estimated in-memory size instead of (or in addition to) key count.

Notes:
- `b/36567833` is a Google-internal Buganizer ref with no public page; this issue tracks it publicly.
- Sibling `PGBKOperation` has the same shape (`max_size = 10 * 1000` elements).
- Sizing accurately is the hard part — `sys.getsizeof` undercounts wrapped/native (e.g. C-extension) accumulators; a coder-based or sampled-RSS estimate is likely needed.

### Issue Priority

Priority: 3 (nice-to-have improvement)

### Issue Components

- Component: Python SDK

Contributor guide

Open the contributing guide

Research direction

Start in sdks/python/apache_beam/runners/worker/operations.py at PGBKCVOperation and its TODO about bounding by in-memory size. Compare the sibling PGBKOperation and inspect how the Count, Mean, min, max, and sum accumulators are represented. Done means the precombine table is bounded using an in-memory-size estimate rather than relying only on key count, while accounting for wrapped or native accumulators.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.