n0-computer / n0-computer/net-tools
netwatch: a single failed rebind strands UdpSocket in Closed permanently (WARN storm at full send rate after macOS sleep/wake)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 29
- Forks
- 36
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 6
Description
Summary
SocketState::rebind() deliberately stays in Closed when the bind fails, with the comment "will retry on next attempt" — but nothing ever arms that retry. UdpSocket::rebind() propagates the error without setting is_broken, and maybe_rebind() — the only recovery path on the send/recv side — is gated on is_broken. So after one failed rebind the socket is stranded in Closed forever:
- every subsequent send hits the
SocketState::Closedarm inpoll_send/try_send, logssocket closedat WARN, and fails withBrokenPipe, - at the caller's full send rate, until the process restarts.
How we hit it
Long-running QUIC daemons (iroh-based) on macOS. On sleep→wake, the interface-change handler fires while interfaces are still coming back up; the rebind attempt fails (address not yet bindable), and the socket never recovers even though the address becomes bindable seconds later.
Observed impact on a dev machine with several such daemons: ~500 netwatch::udp: socket closed WARN events/sec per daemon, sustained indefinitely (4,474 of a 5,000-event log sample). Beyond the daemons' own connectivity loss, the sustained logging/syscall churn coincided with a configd IPMonitor stall that escalated to a watchdogd kernel panic on one occasion — hard to prove causality for that last hop, but the storm is unambiguous and reproducible.
Fix
Mark the socket broken when the rebind fails, so the next send/recv retries the bind via maybe_rebind() and the socket heals as soon as the address is bindable again. A deliberate close() must still stay closed (it never sets is_broken).
PR incoming with the one-line fix plus tests pinning both behaviors (failed-rebind-then-heal, and deliberate-close-stays-closed).
We've been running this patch in production-like daemons for several days: sleep/wake cycles that previously triggered the storm now recover within a second, with zero socket closed events.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing SocketState::rebind(), UdpSocket::rebind(), and maybe_rebind(), then inspect the poll_send and try_send paths described in the issue. Compare failed rebind behavior with deliberate close behavior, and run or extend the tests for failed-rebind-then-heal and deliberate-close-stays-closed. Done means recovery occurs after the address becomes bindable without changing intentional close behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100