n0-computer / n0-computer/net-tools

netwatch: a single failed rebind strands UdpSocket in Closed permanently (WARN storm at full send rate after macOS sleep/wake)

Open
#200 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
29
Forks
36
Avg merge
1d 10h
Merged PRs (30d)
6

Description

Summary

SocketState::rebind() deliberately stays in Closed when the bind fails, with the comment "will retry on next attempt" — but nothing ever arms that retry. UdpSocket::rebind() propagates the error without setting is_broken, and maybe_rebind() — the only recovery path on the send/recv side — is gated on is_broken. So after one failed rebind the socket is stranded in Closed forever:

  • every subsequent send hits the SocketState::Closed arm in poll_send/try_send, logs socket closed at WARN, and fails with BrokenPipe,
  • at the caller's full send rate, until the process restarts.

How we hit it

Long-running QUIC daemons (iroh-based) on macOS. On sleep→wake, the interface-change handler fires while interfaces are still coming back up; the rebind attempt fails (address not yet bindable), and the socket never recovers even though the address becomes bindable seconds later.

Observed impact on a dev machine with several such daemons: ~500 netwatch::udp: socket closed WARN events/sec per daemon, sustained indefinitely (4,474 of a 5,000-event log sample). Beyond the daemons' own connectivity loss, the sustained logging/syscall churn coincided with a configd IPMonitor stall that escalated to a watchdogd kernel panic on one occasion — hard to prove causality for that last hop, but the storm is unambiguous and reproducible.

Fix

Mark the socket broken when the rebind fails, so the next send/recv retries the bind via maybe_rebind() and the socket heals as soon as the address is bindable again. A deliberate close() must still stay closed (it never sets is_broken).

PR incoming with the one-line fix plus tests pinning both behaviors (failed-rebind-then-heal, and deliberate-close-stays-closed).

We've been running this patch in production-like daemons for several days: sleep/wake cycles that previously triggered the storm now recover within a second, with zero socket closed events.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing SocketState::rebind(), UdpSocket::rebind(), and maybe_rebind(), then inspect the poll_send and try_send paths described in the issue. Compare failed rebind behavior with deliberate close behavior, and run or extend the tests for failed-rebind-then-heal and deliberate-close-stays-closed. Done means recovery occurs after the address becomes bindable without changing intentional close behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
networking
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.