apache / apache/pinot

Pinot Dedup requires a lot of memory

Open
#10,571 1 comment 0 reactions 3 assignees Claimed by @chenboat View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

This problem was reported by @ybrankov
Currently, Pinot stream ingestion with exact deduplication requires a lot of memory because it uses [ConcurrentHashMap](https://github.com/apache/pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/PartitionDedupMetadataManager.java#L45).
- Let use say we want to ingest 1K/node and want to keep data for 11 days, we want to be able to deduplicate about 1K/sec * 86400 sec/day * 11 days ~ 1B rows.
- Assuming the primary key is a UUID string, it takes about 36 bytes.
- So, this will require 36 GB just for keys. There is also java overhead and memory needed for storing values.

All this adds up to become too expensive and does not scale well.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.