apache / apache/pinot

Add API to update ZK metadata of a segment

Open
#10,458 11 comments 1 reaction 0 assignees View on GitHub
enhancement good first issue help wanted
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Looking at `BaseSingleSegmentConversionExecutor`, I see that it always uploads the segment, even if the segment did not change but only the metadata (e.g., updating custom map for some minion task ran on the segment). The controller then checks that the segment did not change and only updates ZK.

The network transfer of the segment from minion task to controller seems wasteful in this case, since the minion task should already know that the segment did not change. If there is an API to simply update ZK metadata, then that will be more efficient.

Contributor guide

Open the contributing guide

Research direction

Start at BaseSingleSegmentConversionExecutor and trace the controller path that detects an unchanged segment and updates ZK. Determine how a metadata-only update API should fit between the minion task and controller. Done means unchanged segments can update their ZK metadata without uploading the segment.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.