Add a controller API to reflect whether force commit is finished
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
Force commit is executed asynchronously leveraging the Helix messages. Currently there is no way to tell whether force commit is finished or not. We want to add a controller API to indicate whether all the consuming segments are successfully committed.
There are 2 solutions I can think of:
1. Similar to tracking the reload status, we store the force commit request on the controller in a map, and use an uuid to look up the submitted force commit request. We can store the consuming segments when the force commit request is submitted, and then check if all of them turn into ONLINE state in the ideal state.
2. Do not track anything, and return the smallest of all the last committed segments' creation time. User can use this timestamp to tell whether all the segments are recently committed. For streams with some partition can be ended (e.g. Kinesis), it might need special handling so that the ended partition is excluded.
cc @jadami10 @siddharthteotia @mcvsubbu @sajjad-moradi for some discussion
Contributor guide
Assessment
This issue has not been assessed yet.