apache / apache/beam

Nexmark ratelimiting mode doesn't work

Open
#20,706 0 comments 0 reactions 0 assignees View on GitHub
bug nexmark P3 tests
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

When using the `--isRateLimited` option for Nexmark, the system is supposed to hold events back until they are supposed to be sent, in accordance to the event generations rule.

What happens today is that the events are pushed into the backlog, but are never actually replayed. This leads to only a few events being sent.

When I tested this with the direct runner and flink I saw that the generator got resumed all the time, and when it gets resumed the backlog is not restored. I hacked together a solution for this locally, but even with the queue collected, most events still lack. From my tests only 1/10 make it through.

I don't really know the nexmark code well enough to benchmark anymore beyond this.

Imported from Jira [BEAM-11547](https://issues.apache.org/jira/browse/BEAM-11547). Original Jira may contain additional context.
Reported by: rHermes.

Contributor guide

Open the contributing guide

Research direction

Start at the Nexmark --isRateLimited option and trace how the event generator stores and restores its backlog when resumed. Reproduce the behavior with the Direct Runner and Flink, then verify that events are replayed according to the event generations rule rather than being lost.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.