apache / apache/pinot

[feature] Add Support for Not Dropping Out of Order Events for Upsert Tables

Open
#11,379 2 comments 0 reactions 0 assignees View on GitHub
feature upsert
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

When producers are distributed, it can happen that the time value for two records with the same primary key doesn't match the order of the events in the Kafka topic.

From a user's perspective, particularly for Partial Upsert tables, in many cases they are only interested in applying a set of updates to their table and the order doesn't matter that much.

Given this, it would be great if we could support a config to control whether out of order events should be dropped or not.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how out-of-order events are handled for Partial Upsert tables and where table configuration is defined. The work is complete when a configuration controls whether those events are dropped, with tests covering both enabled and disabled behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
databases, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.