Supporting Consistent Data Push and Data Rollback on Pinot
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
@jackjlli and I have come up with the design document for supporting the consistent push on Pinot. The main idea is to reuse the "segment replacement protocol" added as part of the merge-rollup service to switch old segments with new segments.
The existing protocol has some limitations since it was originally designed in the context of merge/rollup. Here are some main changes:
- Add "reverted" state for segment lineage entry & revert API
- Add "select" API that returns the list of target segments to be replaced given the time range.
- Improve the segment name generator for the unique segment name
- Improve the code to cover some failure cases
https://docs.google.com/document/d/1PUy4wSUPFyEWEW3a88Mipdug3cPj4EpV__lx-BVUTYk/edit#
Contributor guide
Research direction
Start by reading the linked design document and the existing segment replacement protocol from the merge/rollup service. The requested scope includes a reverted lineage state and revert API, a time-range select API, unique segment name generation, and failure-case handling. Done means these design changes are implemented and covered by the relevant existing tests, which the issue does not identify.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100