apache / apache/pinot

Skipping JSON index creation when restarting servers

Open
#12,328 2 comments 0 reactions 0 assignees View on GitHub
feature stale
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

For some of our large tables, reloading the table to generate a JSON index cannot always be done safely or in a timely manner. We prefer to wait until the table's TTL has elapsed and adjust our queries as required based on which segments have the index.

This causes issues when servers are restarted or crash, since during startup they'll discover a missing JSON index on older segments and attempt to create it. This can result in crash looping until the JSON index is removed from the table entirely or enough problematic segments are deleted.

This feature request is to add a server config to skip new index creation when a server starts up.

Contributor guide

Open the contributing guide

Research direction

No file or test is named. Start by tracing server startup when it finds missing JSON indexes on older segments, then identify the server configuration path for disabling index creation. Done means a configuration option prevents new JSON index creation during startup while allowing affected queries to continue according to the available segment indexes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.