Nexmark ratelimiting mode doesn't work
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
When using the `--isRateLimited` option for Nexmark, the system is supposed to hold events back until they are supposed to be sent, in accordance to the event generations rule.
What happens today is that the events are pushed into the backlog, but are never actually replayed. This leads to only a few events being sent.
When I tested this with the direct runner and flink I saw that the generator got resumed all the time, and when it gets resumed the backlog is not restored. I hacked together a solution for this locally, but even with the queue collected, most events still lack. From my tests only 1/10 make it through.
I don't really know the nexmark code well enough to benchmark anymore beyond this.
Imported from Jira [BEAM-11547](https://issues.apache.org/jira/browse/BEAM-11547). Original Jira may contain additional context.
Reported by: rHermes.
Contributor guide
Research direction
Start at the Nexmark --isRateLimited option and trace how the event generator stores and restores its backlog when resumed. Reproduce the behavior with the Direct Runner and Flink, then verify that events are replayed according to the event generations rule rather than being lost.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100