apache / apache/pinot

Manually deleted consuming segments

Open
#9,039 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

PR #8663 adds a new endpoint to recreate consuming segments which have been deleted manually by admins. Now this resume endpoint is used in pause/resume consumption in PR #8986
@mcvsubbu brought up a good point that we can block deleting consuming segments in segment deletion endpoint. If we do this, then periodic RealtimeSegmentValidationManager job - which ensures all partitions are properly consuming - can only fixes the issues that might happen in edge cases of commit protocol and not fix the user errors. This will simplify the code on the periodic job.

Contributor guide

Open the contributing guide

Research direction

Start by reading the segment deletion endpoint and RealtimeSegmentValidationManager, along with PRs #8663 and #8986 for the consuming-segment resume and pause/resume behavior. Determine how manually deleted consuming segments are currently recreated and how deletion should be restricted. Done means user-triggered deletion is prevented where appropriate while the periodic job still handles commit-protocol edge cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
database
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.