apache / apache/pinot

Re-design dedup to not reuse upsert mechanism

Open
#11,045 1 comment 0 reactions 1 assignee Claimed by @atris View on GitHub
enhancement
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Here are some of the main differences between dedup and upsert:
- Dedup is done when ingesting data from the stream (apply to consuming segment only), and no need to track valid docs. The duplicate records are simply dropped
- Dedup window (TTL of the metadata) is a must have to reduce the metadata size
- There is no need to track the record location in the dedup metadata. We do want to track timestamp for the dedup window

One potential solution for the dedup window is to keep 2 rotating maps, each storing metadata for one dedup window, and once the old map is completely out of the dedup window, clear it and use it as the new map.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.