Predicate moves: shrink the commit-abort window (two-phase bulk + delta move)
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 21.8k
- Forks
- 1.6k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 9
Description
Current behavior
For the entire duration of a predicate move, Zero aborts every commit that touches the predicate: movePredicate calls blockTablet before streaming starts and releases it only when the move returns, and the oracle rejects blocked predicates at commit time (dgraph/cmd/zero/oracle.go: "Commits on predicate %s are blocked due to predicate move"). With multi-hour moves of large tablets, that is a multi-hour write outage on the predicate. A failed move also discards all streamed work; there is no resumability.
Proposed direction
Two-phase move:
- Bulk phase: stream the tablet at ts1 without blocking commits.
- Delta phase: block commits, take ts2, stream only keys changed since ts1 (badger's stream framework supports
SinceTs), then flip the tablet to the destination group.
The write-abort window shrinks from O(move duration) to O(delta), minutes instead of hours. An alternative or complement is moving the bulk data outside Raft entirely, the way snapshot streaming already transfers group data.
Progress visibility belongs in this redesign too. MovePredicate is a unary RPC (protos/pb.proto), so Zero is blind while a move runs. A pollable move-status RPC on the source Alpha would let Zero log progress and ETA, and eventually replace the wall-clock timeout with stall detection: cancel only when bytes sent stops advancing.
Context
Surfaced while fixing #9784 (size-aware move timeout). Contained throughput work in the receive path is tracked in #9790; this issue covers the design-level availability fix.
Jira: DGR-309 (https://istari.atlassian.net/browse/DGR-309)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the movePredicate flow and dgraph/cmd/zero/oracle.go to understand blocking and commit handling. Inspect protos/pb.proto, Badger's SinceTs streaming support, and the snapshot streaming path. Done means the design covers bulk and delta phases, progress visibility, and stall-based cancellation without a multi-hour write outage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100