dgraph-io / dgraph-io/dgraph

Predicate moves: shrink the commit-abort window (two-phase bulk + delta move)

Open
#9,791 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
21.8k
Forks
1.6k
Avg merge
2d 5h
Merged PRs (30d)
9

Description

Current behavior

For the entire duration of a predicate move, Zero aborts every commit that touches the predicate: movePredicate calls blockTablet before streaming starts and releases it only when the move returns, and the oracle rejects blocked predicates at commit time (dgraph/cmd/zero/oracle.go: "Commits on predicate %s are blocked due to predicate move"). With multi-hour moves of large tablets, that is a multi-hour write outage on the predicate. A failed move also discards all streamed work; there is no resumability.

Proposed direction

Two-phase move:

  1. Bulk phase: stream the tablet at ts1 without blocking commits.
  2. Delta phase: block commits, take ts2, stream only keys changed since ts1 (badger's stream framework supports SinceTs), then flip the tablet to the destination group.

The write-abort window shrinks from O(move duration) to O(delta), minutes instead of hours. An alternative or complement is moving the bulk data outside Raft entirely, the way snapshot streaming already transfers group data.

Progress visibility belongs in this redesign too. MovePredicate is a unary RPC (protos/pb.proto), so Zero is blind while a move runs. A pollable move-status RPC on the source Alpha would let Zero log progress and ETA, and eventually replace the wall-clock timeout with stall detection: cancel only when bytes sent stops advancing.

Context

Surfaced while fixing #9784 (size-aware move timeout). Contained throughput work in the receive path is tracked in #9790; this issue covers the design-level availability fix.

Jira: DGR-309 (https://istari.atlassian.net/browse/DGR-309)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the movePredicate flow and dgraph/cmd/zero/oracle.go to understand blocking and commit handling. Inspect protos/pb.proto, Badger's SinceTs streaming support, and the snapshot streaming path. Done means the design covers bulk and delta phases, progress visibility, and stall-based cancellation without a multi-hour write outage.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.