Pinot Dedup requires a lot of memory
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
This problem was reported by @ybrankov
Currently, Pinot stream ingestion with exact deduplication requires a lot of memory because it uses [ConcurrentHashMap](https://github.com/apache/pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/PartitionDedupMetadataManager.java#L45).
- Let use say we want to ingest 1K/node and want to keep data for 11 days, we want to be able to deduplicate about 1K/sec * 86400 sec/day * 11 days ~ 1B rows.
- Assuming the primary key is a UUID string, it takes about 36 bytes.
- So, this will require 36 GB just for keys. There is also java overhead and memory needed for storing values.
All this adds up to become too expensive and does not scale well.
Contributor guide
Assessment
This issue has not been assessed yet.