ObolNetwork / ObolNetwork/charon

dkg/pedersen: bundles arriving before protocol start are dropped and never redelivered

Open
#4,682 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

protocol
Dominant language
Go
Stars
222
Forks
138
Avg merge
2d 6h
Merged PRs (30d)
35

Description

Problem

During a replace-operator ceremony (also applies to any pedersen reshare/DKG), a node that starts the protocol a few seconds later than its peers fails the ceremony:

07:31:21.901 ERRO pedersen   Dropping deal bundle, context done       {"from": "16Uiu..."}
07:31:23.144 ERRO pedersen   Dropping deal bundle, context done       {"from": "16Uiu..."}
07:31:23.340 ERRO pedersen   Dropping deal bundle, context done       {"from": "16Uiu..."}
07:31:23.932 INFO cmd        Starting pedersen reshare...
07:31:33.944 WARN pedersen   Sending DKG complaints, deals from these dealers failed verification {"dealer_indices": "[0 1 5]"}
07:31:53.937 ERRO pedersen   [dkg-log Public polynomial missing - evicting dealer 5]
07:31:53.937 ERRO pedersen   [dkg-log Public polynomial missing - evicting dealer 1]
07:31:53.937 ERRO pedersen   [dkg-log Public polynomial missing - evicting dealer 0]
07:31:53.940 ERRO cmd        Application failed to start: ... process-justifications: only 3/5 valid deals - dkg abort

Root cause

Peers ahead in the ceremony broadcast their deal bundles before this node's kyber protocol starts reading Board.IncomingDeal(). The p2p handler pushes into an unbuffered channel (dkg/pedersen/board.go), blocks until the request context expires (5s defaultRcvTimeout), and drops the bundle. The drop is permanent:

  • Sender.SendAsync is fire-and-forget, and the handler acks success even after dropping, so the sender never retransmits.
  • On main, the dedup map records the bundle signature before delivery, so even a retransmit would be refused as a duplicate.

The missing deals then cascade deterministically: complaints against the affected dealers → their justifications cannot be verified without their public polynomials (which were in the dropped deal bundles) → dealers evicted → fewer valid deals than oldThresholddkg abort.

Affected versions

Verified present on main and v1.10.3 (the unbuffered-channel drop with success ack is identical; v1.10.3 predates the dedup map but the sender never retransmits anyway).

Fix

Buffer the deal/response/justification channels to cluster size so bundles arriving before the protocol starts reading are queued instead of dropped, and un-record dropped bundles from the dedup map so a redelivery is accepted.

Workaround

Retry the ceremony with all operators starting as close to simultaneously as possible, and ensure no node is resource-starved (the failing node was ~5-8s behind its peers).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in dkg/pedersen/board.go and trace Board.IncomingDeal(), the p2p handler, Sender.SendAsync, and the dedup map around defaultRcvTimeout. Buffer deal, response, and justification channels to cluster size, ensure dropped bundles can be accepted on redelivery, and verify that bundles arriving before protocol start are not permanently lost.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
blockchain, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
67/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.