apache / apache/pinot

Add sync/async mode when deleting segments

Open
#9,738 0 comments 0 reactions 1 assignee Claimed by @walterddr View on GitHub
feature
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Currently delete segments API delete segments asynchronously (`PinotHelixResourceManager.deleteSegments()`), but delete table API delete segments synchronously (`PinotHelixResourceManager.deleteOfflineTable()` and `PinotHelixResourceManager.deleteRealtimeTable()`). Instead, we should provide a flag so that user can decide how to delete the segments.

When deleting a segment, we delete the segment from ideal state first, then delete ZK metadata and file in deep store. The second step can happen both synchronously and asynchronously. The reason why we want to delete the ZK metadata and deep store file at the same time is because we need the download address from the ZK metadata to point to the deep store file so that it won't left untracked. We should try to delete deep store file first, then delete the ZK metadata to ensure the file is always cleaned up.

To make the process symmetric, when adding the segment, we should first create the ZK metadata, then copy the segment to deep store, in the end add the segment to the ideal state.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.