Support column level APIs for column stats collector in Batch segment creation
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Current segment creation phase, segment stats collection takes the input of a generic row and loops through all the columns, it's good for input files in row format.
For input files in columnar format(parquet), providing an option to collect column stats column by column will reduce memory footprint.
Suggest adding new APIs to 1. collect entries per column basis, 2. generate ColumnIndexCreationInfo for a given column.
cc: @dongxiaoman
Contributor guide
Research direction
Start by locating the batch segment creation phase and its column stats collector, then trace how the current generic-row path loops over columns. Compare this with the parquet or other columnar input path and identify the existing ColumnIndexCreationInfo flow. Done means the collector exposes per-column entry collection and can generate ColumnIndexCreationInfo for one specified column without requiring all columns in memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100