apache / apache/pinot

Add the support of derived column in segment reload when the transform function is defined in the schema config

Open
#6,572 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Recently the transform function of the derived column is moved to `ingestionConfig` in table config, and the transform function definition in schema config is marked as deprecated.

However, it will be good for the back-compatibility to add the support in the segment reload for those derived columns still defined in schema config.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the segment reload path for derived columns and inspect how transform functions are read from the schema config versus ingestionConfig. Done means segment reload supports legacy derived-column transform definitions from schema config while preserving current behavior for ingestionConfig definitions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.