apache / apache/pinot

Enhance the default column handling

Open
#8,699 0 comments 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

When new columns are added to a pinot table, the server is able to automatically generate default columns for the new added columns. There are 2 types of default column that can be auto-generated:
1. New added column without transform function: default column will be filled with default null value (always dictionary encoded)
2. New added column with transform function (derived column): default column will be filled with values calculated based on the transform function

There are 2 things not handled right now:
1. Generate default column raw forward index (not dictionary encoded)
2. Support transform function on another new added column: we need to calculate the dependency between new added columns, and generate them in sequence

See `BaseDefaultColumnHandler` for details

Contributor guide

Open the contributing guide

Research direction

Start by reading BaseDefaultColumnHandler and trace how default columns are generated for newly added Pinot table columns. Define the dependency order for transform functions that reference other new columns, and verify support for raw forward indexes and sequential derived-column generation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.