Predicate moves: pipeline destination Raft proposals and batch KV writes
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 21.8k
- Forks
- 1.6k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 9
Description
Current behavior
The destination side of a predicate move funnels all streamed data through the destination group's Raft, one chunk at a time. batchAndProposeKeyValues (worker/predicate_move.go) accumulates 32MB of KVs, then calls proposeAndWait and blocks until that chunk is replicated to quorum, written to the Raft WAL, and applied on the leader before the next chunk is proposed. The apply path (populateKeyValues -> posting.TxnWriter) opens one badger transaction per KV, so a 32MB chunk of small posting lists commits tens of thousands of individual managed transactions.
Net effect: every byte moved is written roughly 6x on the destination group (Raft WAL plus badger, times 3 replicas), and throughput is bounded by one quorum round trip per 32MB chunk. The report in #9784 works out to about 0.35 MB/s on-disk (20GB in roughly 16 hours).
Proposed change
Two contained optimizations in the receive path:
- Pipeline the proposals. After the initial
CleanPredicatebarrier, chunks are order-independent: badger's stream framework partitions key ranges so keys are disjoint, and every KV is written at the same version (MoveTs). Keep a bounded number of proposals in flight (4-8) and wait for all of them before acking EOF. - Batch the writes. All KVs in a move share the same commit timestamp, so
populateKeyValuescan write them through shared transactions or a write batch instead of oneCommitAtper KV.
Plausibly a 4-8x improvement for changes confined to batchAndProposeKeyValues and populateKeyValues.
Context
Surfaced while fixing #9784 (size-aware move timeout). That fix stops large moves from being cancelled but does not make them faster.
Jira: DGR-308 (https://istari.atlassian.net/browse/DGR-308)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in worker/predicate_move.go with batchAndProposeKeyValues and follow its proposeAndWait flow after the CleanPredicate barrier. Then inspect populateKeyValues and its posting.TxnWriter usage. Done means bounded proposals can remain in flight, all are awaited before EOF is acknowledged, and the shared MoveTs writes avoid one managed transaction per KV.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100