NethermindEth / NethermindEth/pluto

Centralize retry/backoff configuration on `backon`; inject retries where components give up

Open
#618 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement rust
Dominant language
Rust
Stars
8
Forks
5
Avg merge
4d 16h
Merged PRs (30d)
37

Description

Summary

backon is already the workspace retry engine and app/src/retry.rs (the Charon app/retry port) is built on it — but the workspace has drifted into at least 5 distinct backoff parameter sets and 4 drive mechanisms:

  • retry.rs defaults (250ms/12s/1.6) — zero call sites (#534 tracks wiring it into the duty callbacks).
  • core::expbackoff fast() (100ms/5s) and default() (1s/120s) — used by scheduler/sse/bootnode, sometimes driven manually via .next() + expect.
  • p2p/src/relay/dial.rs#L104-L126 — a hand-written duplicate of expbackoff::default() (its doc even cites the same Charon config), reimplemented because p2p needs a pollable Duration rather than an async wrapper.
  • quic_upgrade.rs — a third scheme in units of minutes (1→512, doubling).
  • eth1wrap — Alloy's RetryBackoffLayer::new(10, 1000, 100), a fourth policy in a foreign library's units.
  • Fixed-delay loops in dkg/sync (250ms), cli test/peers (for attempt in 0..5, 5s intervals). Same family: dkg busy-polls node signatures on a 100ms ticker, re-locking and cloning the accumulated slot vector every tick (nodesigs.rs#L144-L164), and exchanger.rs#L561 polls with a bare 100ms sleep — both want a Notify/watch signal instead of a poll.

Meanwhile, components that should retry just give up and wait for the next tick: scheduler resolve_duties errors are logged "(retrying next slot)" — a beacon-node blip loses a slot's duty resolution; bcast/recast retries next epoch; the wire-layer store/broadcast sinks log and swallow errors (wire.rs#L739-L746 and siblings) — precisely the points Charon wraps in async-retry.

Proposed change

  • One retry module (the existing app::retry + core::expbackoff, merged or clearly layered) exposing the named policies (fast, default, plus a pollable-Duration helper for poll-based behaviours so relay/dial.rs can delete its copy).
  • Replace the manual .next()/hand-rolled loops with backon's Retryable or the shared pollable helper; express eth1wrap's policy in the same config vocabulary.
  • Inject retries at the give-up sites above (scheduler duty resolution, recast, wire sinks), with per-duty deadlines from the existing DeadlineCalculator plumbing. #534 covers the five Charon duty-callback wrap points; this issue covers the config unification and the remaining sites.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with crates/app/src/retry.rs and crates/core/src/expbackoff.rs, then trace the listed scheduler, relay/dial.rs, quic_upgrade.rs, eth1wrap, dkg, recast, and wire sink entry points. Compare their retry and polling behavior before deciding how the shared policies should be layered. Done means the duplicated configurations and manual loops are addressed, and the named give-up sites retry with the intended deadlines.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.