apache / apache/fluss

Primary key table support standby replica

Open
#1,990 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

Primary key table support standby replica.

During testing, I found that the primary reason for the slow recovery of PrimaryKey tables is the slow download of snapshot files, which accounts for about 90% of the total recovery time, while the apply cdc log phase takes up only a small fraction.

Therefore, could we introduce a standby replica that continuously synchronizes the latest snapshots? During recovery, we could restore directly from this standby replica.

This approach might even eliminate the need for rebalancing of PrimaryKey table.

### Solution

_No response_

### Anything else?

_No response_

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by investigating PrimaryKey table recovery, snapshot-file downloads, CDC log application, and rebalancing in the Fluss codebase. Define how a standby replica would continuously synchronize snapshots and how recovery would restore from it; done means the design addresses recovery speed and the proposed reduction in rebalancing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.