apache / apache/pinot

Add a controller API to reflect whether force commit is finished

Open
#10,069 4 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Force commit is executed asynchronously leveraging the Helix messages. Currently there is no way to tell whether force commit is finished or not. We want to add a controller API to indicate whether all the consuming segments are successfully committed.

There are 2 solutions I can think of:
1. Similar to tracking the reload status, we store the force commit request on the controller in a map, and use an uuid to look up the submitted force commit request. We can store the consuming segments when the force commit request is submitted, and then check if all of them turn into ONLINE state in the ideal state.
2. Do not track anything, and return the smallest of all the last committed segments' creation time. User can use this timestamp to tell whether all the segments are recently committed. For streams with some partition can be ended (e.g. Kinesis), it might need special handling so that the ended partition is excluded.

cc @jadami10 @siddharthteotia @mcvsubbu @sajjad-moradi for some discussion

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.