apache / apache/pinot

Updating JSON index config

Open
#10,506 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

Let’s say I create a segment with the JSON index with default params and I decide afterwards that I want to exclude some fields.

At the moment there isn't a way to reload the segment because that code path checks if a JSON index exists on that column (which it does) and then skips over it.

(https://github.com/apache/pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/JsonIndexHandler.java#L89)

I am able to replace the JSON index if I first completely remove it from the table config, reload segments, then add it back to the table config, and reload all segments.

But it would be cool if we could do that in one step rather than two.

cc @Jackie-Jiang

Contributor guide

Open the contributing guide

Research direction

Start in pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/JsonIndexHandler.java at the referenced line, then trace the segment reload path and its table-config checks. Confirm how an existing JSON index is handled when its excluded fields change. Done means updated JSON index configuration can replace the existing index during one reload operation, without removing and re-adding it in the table config.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.