apache / apache/pinot

Supporting Consistent Data Push and Data Rollback on Pinot

Open
#7,813 1 comment 5 reactions 0 assignees View on GitHub
design-review
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

@jackjlli and I have come up with the design document for supporting the consistent push on Pinot. The main idea is to reuse the "segment replacement protocol" added as part of the merge-rollup service to switch old segments with new segments.

The existing protocol has some limitations since it was originally designed in the context of merge/rollup. Here are some main changes:

- Add "reverted" state for segment lineage entry & revert API
- Add "select" API that returns the list of target segments to be replaced given the time range.
- Improve the segment name generator for the unique segment name
- Improve the code to cover some failure cases

https://docs.google.com/document/d/1PUy4wSUPFyEWEW3a88Mipdug3cPj4EpV__lx-BVUTYk/edit#

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked design document and the existing segment replacement protocol from the merge/rollup service. The requested scope includes a reverted lineage state and revert API, a time-range select API, unique segment name generation, and failure-case handling. Done means these design changes are implemented and covered by the relevant existing tests, which the issue does not identify.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.