apache / apache/beam

Improve scalability of the Watch transform

Open
#18,459 2 comments 0 reactions 0 assignees View on GitHub
bug core java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

[https://github.com/apache/beam/pull/3565](https://github.com/apache/beam/pull/3565) introduces the Watch transform [http://s.apache.org/beam-watch-transform](http://s.apache.org/beam-watch-transform).

The implementation leaves several scalability-related TODOs:
1) The state stores hashes and timestamps of outputs that have already been output and should be omitted from future polls. We could garbage-collect this state, e.g. dropping elements from "completed" and from addNewAsPending() if their timestamp is more than X behind the watermark.
2) When a poll returns a huge number of elements, we don't necessarily have to add all of them into state.pending - instead we could add only N oldest elements and ignore others, relying on future poll rounds to provide them, in order to avoid blowing up the state. Combined with garbage collection of GrowthState.completed, this would make the transform scalable to very large poll results.

Imported from Jira [BEAM-2680](https://issues.apache.org/jira/browse/BEAM-2680). Original Jira may contain additional context.
Reported by: jkff.

Contributor guide

Open the contributing guide

Research direction

Start with the Watch transform introduced by PR 3565 and review the additional context in Jira BEAM-2680. Determine how completed and pending state is managed; done means stale completed entries can be garbage-collected and large poll results no longer require retaining every pending element while preserving future results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance, stream-processing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.