[feature] Add Support for Not Dropping Out of Order Events for Upsert Tables
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
When producers are distributed, it can happen that the time value for two records with the same primary key doesn't match the order of the events in the Kafka topic.
From a user's perspective, particularly for Partial Upsert tables, in many cases they are only interested in applying a set of updates to their table and the order doesn't matter that much.
Given this, it would be great if we could support a config to control whether out of order events should be dropped or not.
Contributor guide
Research direction
Start by tracing how out-of-order events are handled for Partial Upsert tables and where table configuration is defined. The work is complete when a configuration controls whether those events are dropped, with tests covering both enabled and disabled behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- databases, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100