apache / apache/pinot

should SegmentDeletionManager expose the 2 public APIs?

Open
#8,092 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

Currently SegmentDeletionManager has
1. a `deleteSegments` API that allows users to delete segments from property store and from deep store.
2. a `removeSegmentsFromStore` API that allows users to only delete from deep store.

This creates confusion regarding failure recovery, when the component service dies in the middle of the execution, they can either result in
- using `deleteSegments`, property store could've deleted segment A, but file still exist in deep store
- using `removeSegmentsFromStore`, segment A's file could've been deleted from deep store, but property store could still have segment A.

Which one should we go with as failure recovery strategy?
From the point that we have a retention manager that periodically checks deep store for files to delete, I think we should always first delete from property store and next delete from deep store. thoughts?

Contributor guide

Open the contributing guide

Research direction

Start by reading SegmentDeletionManager and its deleteSegments and removeSegmentsFromStore APIs, then trace the retention manager's periodic deep-store cleanup. Compare the failure-recovery behavior when property-store and deep-store deletion occur in different orders. Done means agreeing on the public API and documenting or implementing the chosen recovery strategy.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.