Enhance the default column handling
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
When new columns are added to a pinot table, the server is able to automatically generate default columns for the new added columns. There are 2 types of default column that can be auto-generated:
1. New added column without transform function: default column will be filled with default null value (always dictionary encoded)
2. New added column with transform function (derived column): default column will be filled with values calculated based on the transform function
There are 2 things not handled right now:
1. Generate default column raw forward index (not dictionary encoded)
2. Support transform function on another new added column: we need to calculate the dependency between new added columns, and generate them in sequence
See `BaseDefaultColumnHandler` for details
Contributor guide
Research direction
Start by reading BaseDefaultColumnHandler and trace how default columns are generated for newly added Pinot table columns. Define the dependency order for transform functions that reference other new columns, and verify support for raw forward indexes and sequential derived-column generation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100