kvcache-ai / kvcache-ai/AgentENV

fix(p2p): sandbox veth churn kills the iroh endpoint ~1.3s after startup, disabling P2P for the process lifetime

Open
#138 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.5k
Forks
309
Avg merge
1d 9h
Merged PRs (30d)
66

Description

## Summary

With `[p2p].enabled = true`, the iroh endpoint and the `iroh-blobs` store are torn
down **~1.3 s after node startup, before a single artifact is ever served**. On an
8-node cluster, 7 of 8 nodes died this way; the 8th survived only by winning a race.
Once dead, nothing ever restarts the transport, so P2P is silently unavailable for
the entire lifetime of the process and every overlaybd read falls back to the origin
registry.

The trigger is AgentENV's own warm-pool priming: the `veth-*` interfaces it creates
in the runtime pod's network namespace are seen by `netwatch` as a *major* link
change, which makes iroh rebind its UDP socket — and that rebind is not recoverable
when it fails.

## Environment

| | |
|---|---|
| AgentENV runtime | `v0.1.3` |
| iroh | `1.0.0-rc.0` (also reproduces by inspection on `1.0.3`) |
| netwatch | `0.19.1` (unmodified from crates.io) |
| Kernel | 6.8, cgroup v2 |
| Deployment | Kubernetes DaemonSet, pod has its **own** netns (`hostNetwork` unset) |
| Config | `[p2p] enabled=true, transport="iroh", listen_addr="0.0.0.0:0"`, `[snapshot] p2p_enabled=true`, `[ublk] download_enable=true` |
| Scale | 8 runtime nodes, all with identical config |

## Reproduction

Enable P2P with the default `listen_addr = "0.0.0.0:0"` and start the node. The
failing log lines are **debug-level only**, so an `info` log shows nothing at all —
this is why the failure went unnoticed. Use:

```
RUST_LOG=info,iroh=debug,iroh_blobs=debug,netwatch=debug,noq=debug
```

Observed on 7 of 8 nodes, marker counts byte-identical on every one of them
(`rebinding`=1, `Address already in use`=3, `socket closed`=2, `closing database`=1):

```
13.572 iroh::socket::transports::ip: binding addr=0.0.0.0:0
13.572 iroh::socket::transports::ip: successfully bound local_addr=0.0.0.0:51871
13.597 iroh::_events::direct_addrs: addrs={10.126.17.156:51871 Local}
13.597 iroh::endpoint: iroh endpoint bound iroh_version=1.0.0-rc.0
13.607 agentenv::p2p::iroh::transport: iroh artifact transport started
13.760 agentenv::sandbox::firecracker::pool: priming firecracker pool low_watermark=2
14.723 agentenv::sandbox::firecracker::pool: firecracker pool primed warm=2 elapsed_ms=962
14.794 iroh::_events::link_change: state=State { interfaces: {...} }
14.795 iroh::socket: link change detected is_major=true
14.795 netwatch::udp: rebinding 0.0.0.0:51871
14.795 netwatch::udp: rebind failed, will retry on next attempt: Address already in use (os error 98)
14.795 WARN iroh::socket::transports: failed to rebind Os { code: 98, kind: AddrInUse }
14.795 WARN iroh::socket: failed to rebind transports: Os { code: 98, kind: AddrInUse }
14.795 iroh::_events::direct_addrs: addrs={10.12.0.2:51871, 10.12.0.4:51871, 10.126.17.156:51871}
14.795 WARN netwatch::udp: socket closed
14.795 ERROR noq::endpoint: I/O error: socket closed
14.796 iroh_blobs::store::fs::meta: closing database
14.799 iroh::protocol: Shutting down remaining tasks
```

After this point the process logs no further iroh activity for its whole lifetime.

## Root cause

Three separate defects compose into the failure. Only the first is AgentENV's.

### 1. Sandbox `veth` churn is reported as a major link change (AgentENV + netwatch)

`10.12.0.2` and `10.12.0.4` are host-side ends of sandbox veth pairs, inside
`[network].veth_cidr = 10.12.0.0/16`. They appear in `direct_addrs` in the *same
millisecond* as `is_major=true`, immediately after `firecracker pool primed warm=2`.
So AgentENV's own routine warm-pool priming is what knocks out its own P2P transport.

On Linux, netwatch treats every interface as relevant
(`netwatch-0.19.1/src/netmon/linux.rs`):

```rust
pub(crate) fn is_interesting_interface(_name: &str) -> bool {
true
}
```

Sandbox veths are node-local plumbing and can never carry P2P traffic, so they should
not influence the endpoint's address set or force a rebind. On a busy node these are
created and destroyed constantly, so this is not a one-shot startup problem — it is a
permanent source of endpoint churn.

### 2. `netwatch`'s rebind destroys a working socket before it knows it can replace it (upstream)

`netwatch-0.19.1/src/udp.rs` (verified byte-identical to the crates.io release):

```rust
fn rebind(&mut self) -> io::Result<()> {
let addr = match self { /* resolved local addr, i.e. 0.0.0.0:51871, not 0.0.0.0:0 */ };
debug!("rebinding {}", addr);

// Transition to Closed first to drop the old socket.
// This is needed so the port is released before we try to bind again.
if let Self::Connected { state, .. } = self {
*self = SocketState::Closed { addr, .. }; // <-- working socket dropped here
}

match Self::bind(addr) {
Ok(new_state) => { *self = new_state; Ok(()) }
Err(err) => {
// Stay in Closed state but allow future rebind attempts
debug!("rebind failed, will retry on next attempt: {}", err);
Err(err)
}
}
}
```

Two problems:

- The healthy socket is dropped **before** a replacement is known to be obtainable.
Any transient bind failure therefore costs the endpoint permanently.
- The "will retry on next attempt" comment is not honoured in practice. No retry ever
fires in our logs — a single failure is terminal. The next `is_major` link change
would be needed to try again, and none arrives, because once the warm pool is primed
the interface set stops changing.

The resulting `Closed` state makes every subsequent send/recv return
`BrokenPipe: socket closed`, which `noq` escalates to `I/O error: socket closed`, which
in turn closes the `iroh-blobs` store.

### 3. The `EADDRINUSE` is a race, and AgentENV never notices the endpoint is gone

One node out of eight ran the identical sequence and its rebind *succeeded*; its
endpoint is still healthy hours later. So this is a timing race on re-binding a
just-released ephemeral port (the pod's range is 32768–60999), not a structural
conflict.

We ruled out a duplicated/retained fd inside the stack:

- `netwatch/src/udp.rs` exposes no fd-escaping API (no `as_raw_fd`, `into_inner`,
`dup`, `ManuallyDrop`, `mem::forget`).
- `noq_udp::UdpSockRef<'a>` is `socket2::SockRef<'a>` — a borrow. `UdpSocketState`
stores no fd.
- `iroh/src/socket/transports/ip.rs:277` only delegates: `self.socket.rebind()?`.
- iroh binds exactly one socket; there is no second (v6) socket to conflict with.

Independently of who wins the port race, AgentENV has no supervision of the P2P
transport: a dead endpoint is never detected, never reported, and never restarted.
A `p2p.enabled = true` node that lost its endpoint is indistinguishable from a healthy
one from the outside, and simply serves every read from the origin registry.

## Suggested fixes

1. **Do not let sandbox veths drive endpoint rebinds.** Filter AgentENV's own
`veth-*` prefix out of netwatch's interface interest set (and out of the reported
address set), so warm-pool and sandbox churn is invisible to iroh.
2. **Make rebind non-destructive.** Retry the same port a small number of times, and
if it stays unavailable, fall back to binding port `0`. A fresh ephemeral port is
strictly better than a dead endpoint — address changes are exactly what the rebind
path is designed to handle. Worth reporting upstream to `netwatch`/`iroh`.
3. **Supervise the transport.** Surface endpoint liveness (heartbeat / node metrics)
and re-create the endpoint when it dies, so this can never again be a silent
degradation.

## Second, independent finding: the overlaybd layer catalog is never populated

Even with a healthy endpoint, the overlaybd P2P path cannot hit. Across all 8 nodes
there has never been a single `overlaybd-layer/v1/sha256:*` key in the iroh catalog
(the current catalog WAL is 0 bytes on every node; the only keys present anywhere are
`snapshot/v1/artifacts/*`). Three reasons, all in code:

- The only publisher of `overlaybd-layer/v1/*` is `publish_completed_layer()`
(`storage/overlaybd/src/bk_download.rs:275`, invoked at `:736` and `:770`), which
runs **only after a complete background layer download**. Locally complete commits
that need no download are also gated behind the same path.
- `DownloadConfig::default()` (`storage/overlaybd/src/config.rs:116-129`) sets
`delay: 300, delay_extra: 30`, inherited verbatim by `src/setup/deps.rs:286-288` via
`..Default::default()`. So no background download even *starts* for 300–330 s, and
any sandbox shorter-lived than that publishes nothing, ever.
- Meanwhile the descriptor miss TTL is 5 s (`src/overlaybd/p2p/facade.rs:32-33`), so an
empty catalog means a fresh full lookup every 5 s **on the block-read critical
path**, with `lookup_peers` handshaking peers sequentially
(`src/p2p/iroh/transport.rs:218-249`, carrying `// TODO: parallelize lookups to
multiple peers.`). This is the mechanism behind the measured exec p50 of 28.7 ms
(P2P off) vs 2766.7 ms (P2P on, empty catalog).

Suggested fix: publish already-complete local layers immediately instead of only as a
side effect of a delayed background download, and avoid broad peer polling when the
scheduler artifact index returns no hint. (Lookup fan-out itself is tracked in #95.)

Contributor guide

Open the contributing guide

Research direction

Start with src/p2p/iroh/transport.rs, storage/overlaybd/src/bk_download.rs, src/setup/deps.rs, and src/overlaybd/p2p/facade.rs; inspect the referenced netwatch and iroh rebind paths. Reproduce with the supplied RUST_LOG settings while priming the Firecracker pool. Done requires preventing sandbox veth churn from killing P2P, handling endpoint failure visibly, and ensuring complete local layers can populate the catalog.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, rust
Domain
backend, distributed-systems, networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.