apache / apache/pinot

Pinot can't skip creating derived column when configured in transform functions

Open
#7,627 0 comments 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

When reloading segments, pinot will skip many segments when derived column not configured in schema or segments metadata , which will cause some segments not synced latest new columns

We got the following error message when query from table:
![image](https://user-images.githubusercontent.com/13634486/138628158-fe6abe65-7572-4130-9e73-0d0ff05ba96d.png)

Code need remove:
```
for (String argument : arguments) {
ColumnMetadata columnMetadata = _segmentMetadata.getColumnMetadataFor(argument);
if (columnMetadata == null) {
LOGGER.warn("Skip creating derived column: {} because argument: {} does not exist in the segment", column,
argument);
return false;
}
argumentsMetadata.add(columnMetadata);
}
```

Contributor guide

Open the contributing guide

Research direction

Search the Pinot codebase for the shown "Skip creating derived column" warning and inspect the surrounding segment-reload logic. Remove the indicated early-return behavior, then verify that reloaded segments create derived columns from transform functions and that queries no longer encounter missing latest columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.