Re-design dedup to not reuse upsert mechanism
Open
enhancement
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
Here are some of the main differences between dedup and upsert:
- Dedup is done when ingesting data from the stream (apply to consuming segment only), and no need to track valid docs. The duplicate records are simply dropped
- Dedup window (TTL of the metadata) is a must have to reduce the metadata size
- There is no need to track the record location in the dedup metadata. We do want to track timestamp for the dedup window
One potential solution for the dedup window is to keep 2 rotating maps, each storing metadata for one dedup window, and once the old map is completely out of the dedup window, clear it and use it as the new map.
Contributor guide
Assessment
This issue has not been assessed yet.