Enable parallel predicate moves across Alpha groups to remove single-threaded constraint and accelerate cluster reorganization
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 21.8k
- Forks
- 1.6k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 9
Description
Problem Statement
Dgraph Zero currently enforces a strict global lock allowing only one predicate move at a time across the entire cluster. When rebalancing or reorganizing clusters containing dozens or hundreds of predicates, processing them in a strictly serial, non-concurrent fashion is prohibitively time-consuming, taking days or even weeks to complete.
Impact
- Operational Bottlenecks: Cluster maintenance, node decommissioning, and rebalancing operations are blocked by single-file predicate transfers.
- Prohibitive Execution Times: Even if individual predicate moves are optimized, transferring multiple predicates sequentially creates unacceptable operational overhead and unnecessarily long maintenance windows. Writes to the predicate are fully blocked until the predicate moves. In a schema with lots of interconnection, this block expands to large amounts of predicates, causing unknown transaction issues to users. Directly, this error is never surfaced to the client. (See PR starting to address this: https://github.com/dgraph-io/dgraph/pull/9747)
Desired Outcome
Dgraph Zero should support concurrent predicate moves across the cluster, allowing operators to move multiple predicates in parallel while maintaining cluster stability and data integrity.
Acceptance Criteria
- Multiple predicate moves can execute concurrently across the cluster.
- Total cluster rebalancing duration is significantly reduced compared to the single-threaded serial baseline.
- Data integrity and tablet ownership state remain consistent during concurrent moves.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing PR #9747 and the Dgraph Zero predicate-move coordination described in this issue. Map how moves are serialized and how tablet ownership and writes are protected before proposing a concurrency design. Done means multiple moves run safely in parallel, preserve data integrity, and reduce rebalancing time.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100