ObolNetwork / ObolNetwork/charon

dkg: ceremony aborts on single transient p2p send failure

Open
#4,685 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug protocol
Dominant language
Go
Stars
222
Forks
138
Avg merge
2d 6h
Merged PRs (30d)
35

Description

Problem

During a `charon alpha edit replace-operator` ceremony, one operator's node completed the Pedersen reshare successfully but then crashed during the lock-hash partial signature exchange:

ERRO cmd Application failed to start: run replace operator protocol: write message: i/o deadline reached {"protocol": "/charon/parsigex/2.0.0"}
    p2p/sender.go:361 .Send
    core/parsigex/parsigex.go:151 .Broadcast
    ...

A single stalled stream write to one peer (likely a dying relay circuit) hit the send deadline and aborted the entire ceremony for everyone.

Root causes

  1. No retries: all DKG transports (`parsigex` exchange, `bcast`, pedersen board, frost round 1) abort on the first failed send, even though ceremony receivers deduplicate messages and a retry would be safe.
  2. No peer in the error: `p2p.Send`/`SendReceive` errors only name the protocol, so the unreachable peer could not be identified from logs.

Related observations from the incident (not addressed by the fix PR):

  • On all released versions (≤ v1.9.5), the other operators hung indefinitely on "Pedersen reshare completed" waiting for the crashed node's signatures — #4616 (unreleased) adds the fail-fast timeout.
  • `ParSigEx` passes its send/receive options only to the receive-side handler, so the DKG's `p2p.WithSendTimeout(--timeout)` is silently dropped and sends run with the hardcoded 7s default deadline — which matches the incident timeline exactly.

Impact

Any transient p2p blip during a DKG or cluster mutation ceremony (dkg, add/remove/replace operator, reshare) fails the whole ceremony, requiring all operators to coordinate a rerun — and on current releases, leaves the remaining operators hung without an error.

No artifacts were written anywhere (writes happen only after all exchanges complete), so the incident was safe to rerun.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at p2p/sender.go:361 and core/parsigex/parsigex.go:151, then trace the parsigex, bcast, Pedersen board, and FROST round 1 send paths. Check how send options reach the receive-side handler and how ceremony receivers deduplicate messages. Done means transient sends can be retried safely and send errors identify the unreachable peer.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.