Consider adding support for batching state requests
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 204
Description
The current user facing API looks like a future which can be resolved later, we want to minimize the amount of time a user spends blocking after queuing up all their requests.
Google Cloud Dataflow has experienced performance problems in the past without having support for a batch API because solutions that add fixed delays ("wait for N milliseconds for a batch to form") have caused large problems for pipelines with few numbers of keys.
Imported from Jira [BEAM-7000](https://issues.apache.org/jira/browse/BEAM-7000). Original Jira may contain additional context.
Reported by: lcwik.
Subtask of issue #18492
Contributor guide
Research direction
Start with parent issue #18492 and the original Jira ticket BEAM-7000, since this subtask names no files, tests, or entry points. Define the batching API and its completion behavior so queued state requests can be handled without fixed delays, then verify the performance concern described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100