earlyMessageRejectionPeriod can drop messages unexpectedly when tasks run long
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
To ensure that replica Kafka tasks are making decisions in sync with each other, earlyMessageRejectionPeriod works by setting a maximumMessageTime on the Kafka tasks spawned by a supervisor, based on the configured taskDuration of the task. However, in some situations, tasks can run longer than their configured duration:
- The remainingSeconds counter resets when supervisors fail over
- Supervisors might not be running when the task duration arrives
- Replica tasks may continue running for some time after the task duration arrives, due to the checkpoint-and-sync protocol
In these situations, if earlyMessageRejectionPeriod is too low, messages will be dropped unexpectedly.
I'm not sure what the best way is to fix this -- we do want a fixed maximumMessageTime in order to ensure that all replicas are in sync.
Contributor guide
Research direction
Start by tracing how earlyMessageRejectionPeriod sets maximumMessageTime for Kafka tasks spawned by a supervisor, then examine the remainingSeconds reset, supervisor downtime, and checkpoint-and-sync cases described in the issue. Define a solution that preserves replica synchronization without unexpectedly dropping messages when tasks outlive their configured duration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100