ObolNetwork / ObolNetwork/charon
dkg: ceremony aborts on single transient p2p send failure
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 222
- Forks
- 138
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 35
Description
Problem
During a `charon alpha edit replace-operator` ceremony, one operator's node completed the Pedersen reshare successfully but then crashed during the lock-hash partial signature exchange:
ERRO cmd Application failed to start: run replace operator protocol: write message: i/o deadline reached {"protocol": "/charon/parsigex/2.0.0"}
p2p/sender.go:361 .Send
core/parsigex/parsigex.go:151 .Broadcast
...
A single stalled stream write to one peer (likely a dying relay circuit) hit the send deadline and aborted the entire ceremony for everyone.
Root causes
- No retries: all DKG transports (`parsigex` exchange, `bcast`, pedersen board, frost round 1) abort on the first failed send, even though ceremony receivers deduplicate messages and a retry would be safe.
- No peer in the error: `p2p.Send`/`SendReceive` errors only name the protocol, so the unreachable peer could not be identified from logs.
Related observations from the incident (not addressed by the fix PR):
- On all released versions (≤ v1.9.5), the other operators hung indefinitely on "Pedersen reshare completed" waiting for the crashed node's signatures — #4616 (unreleased) adds the fail-fast timeout.
- `ParSigEx` passes its send/receive options only to the receive-side handler, so the DKG's `p2p.WithSendTimeout(--timeout)` is silently dropped and sends run with the hardcoded 7s default deadline — which matches the incident timeline exactly.
Impact
Any transient p2p blip during a DKG or cluster mutation ceremony (dkg, add/remove/replace operator, reshare) fails the whole ceremony, requiring all operators to coordinate a rerun — and on current releases, leaves the remaining operators hung without an error.
No artifacts were written anywhere (writes happen only after all exchanges complete), so the incident was safe to rerun.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at p2p/sender.go:361 and core/parsigex/parsigex.go:151, then trace the parsigex, bcast, Pedersen board, and FROST round 1 send paths. Check how send options reach the receive-side handler and how ceremony receivers deduplicate messages. Done means transient sends can be retried safely and send errors identify the unreachable peer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100