apache / apache/pinot

Allow to add derived columns that depends on new columns

Open
#12,186 3 comments 0 reactions 0 assignees View on GitHub
stale
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Right now Pinot fails when a table is modified in such a way that two columns are added and one of them depend on the other.

More specifically, assuming table T has columns A1, A2 and A3, it should be possible to add columns B and C such as:

```js
//table config
"ingestionConfig": {
"transformConfigs": [
{
"columnName": "C",
"transformFunction": "B + 1"
}
]
}
```

But right now this fails with:
```
Caused by: java.lang.RuntimeException: Failed to create derived column: C because argument: B does not exist in the segment
```

This error is thrown in https://github.com/apache/pinot/blob/85b5779cff8d33412937ceb6d4dfc972b6c8bd97/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java#L374 when at least one of the columns used in the transform function is not included in the columns already stored in the segment.

This is not a blocker because the same change can be done by first adding column B, reload and then modify the table again to add column D and reload.

It is important to note that at the same time we add this feature we should add checks to avoid cycle dependencies. Specifically, it should be forbidden to add columns `E1, E2, ..., En` when `E1` depends on `En` and `Ek` depends on `E(k-1)` for all k in [1, n].

It is probably not enough to remove the check in BaseDefaultColumnHandler, we also need to verify that the code that actually computes the value of the derived column actually works.

Contributor guide

Open the contributing guide

Research direction

Start in pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java at the reported derived-column check, then trace the code that computes derived-column values during segment loading. Verify that multiple new transform columns are evaluated in dependency order, reject cyclic dependencies, and confirm the existing single-column behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.