Allow segment generation with a bounded memory
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Current segment creation requires unbounded memory as we use hashmap in AbstractColumnStatisticsCollector implementations.
This may fail minion tasks also future connectors like flink/spark/presto data sink.
The goal here is to allow configure a bounded memory size for segment creation.
@Jackie-Jiang
Contributor guide
Research direction
Start by reviewing the AbstractColumnStatisticsCollector implementations and the segment creation path to understand how their hashmaps grow. Determine how a configurable bounded-memory strategy should apply to segment creation, including minion tasks and future Flink, Spark, or Presto data sinks. Done means segment creation no longer requires unbounded memory and the chosen bound is configurable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100