apache / apache/pinot

Allow segment generation with a bounded memory

Open
#6,422 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Current segment creation requires unbounded memory as we use hashmap in AbstractColumnStatisticsCollector implementations.

This may fail minion tasks also future connectors like flink/spark/presto data sink.

The goal here is to allow configure a bounded memory size for segment creation.

@Jackie-Jiang

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the AbstractColumnStatisticsCollector implementations and the segment creation path to understand how their hashmaps grow. Determine how a configurable bounded-memory strategy should apply to segment creation, including minion tasks and future Flink, Spark, or Presto data sinks. Done means segment creation no longer requires unbounded memory and the chosen bound is configurable.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.