dgraph-io / dgraph-io/dgraph

Predicate moves: pipeline destination Raft proposals and batch KV writes

Open
#9,790 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
21.8k
Forks
1.6k
Avg merge
2d 5h
Merged PRs (30d)
9

Description

Current behavior

The destination side of a predicate move funnels all streamed data through the destination group's Raft, one chunk at a time. batchAndProposeKeyValues (worker/predicate_move.go) accumulates 32MB of KVs, then calls proposeAndWait and blocks until that chunk is replicated to quorum, written to the Raft WAL, and applied on the leader before the next chunk is proposed. The apply path (populateKeyValues -> posting.TxnWriter) opens one badger transaction per KV, so a 32MB chunk of small posting lists commits tens of thousands of individual managed transactions.

Net effect: every byte moved is written roughly 6x on the destination group (Raft WAL plus badger, times 3 replicas), and throughput is bounded by one quorum round trip per 32MB chunk. The report in #9784 works out to about 0.35 MB/s on-disk (20GB in roughly 16 hours).

Proposed change

Two contained optimizations in the receive path:

  1. Pipeline the proposals. After the initial CleanPredicate barrier, chunks are order-independent: badger's stream framework partitions key ranges so keys are disjoint, and every KV is written at the same version (MoveTs). Keep a bounded number of proposals in flight (4-8) and wait for all of them before acking EOF.
  2. Batch the writes. All KVs in a move share the same commit timestamp, so populateKeyValues can write them through shared transactions or a write batch instead of one CommitAt per KV.

Plausibly a 4-8x improvement for changes confined to batchAndProposeKeyValues and populateKeyValues.

Context

Surfaced while fixing #9784 (size-aware move timeout). That fix stops large moves from being cancelled but does not make them faster.

Jira: DGR-308 (https://istari.atlassian.net/browse/DGR-308)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in worker/predicate_move.go with batchAndProposeKeyValues and follow its proposeAndWait flow after the CleanPredicate barrier. Then inspect populateKeyValues and its posting.TxnWriter usage. Done means bounded proposals can remain in flight, all are awaited before EOF is acknowledged, and the shared MoveTs writes avoid one managed transaction per KV.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.