apache / apache/pinot

Support column level APIs for column stats collector in Batch segment creation

Open
#6,267 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Current segment creation phase, segment stats collection takes the input of a generic row and loops through all the columns, it's good for input files in row format.

For input files in columnar format(parquet), providing an option to collect column stats column by column will reduce memory footprint.

Suggest adding new APIs to 1. collect entries per column basis, 2. generate ColumnIndexCreationInfo for a given column.

cc: @dongxiaoman

Contributor guide

Open the contributing guide

Research direction

Start by locating the batch segment creation phase and its column stats collector, then trace how the current generic-row path loops over columns. Compare this with the parquet or other columnar input path and identify the existing ColumnIndexCreationInfo flow. Done means the collector exposes per-column entry collection and can generate ColumnIndexCreationInfo for one specified column without requiring all columns in memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.