network start hangs forever when the FSEvents notification for pocketic.port is missed — port discovery has no polling fallback or timeout

Summary

icp network start can park forever with the launcher alive at 0% CPU, an empty --status-dir, and its pocket-ic child healthy and listening — because the launcher learns pocket-ic's port only through a notify (FSEvents) watcher callback on the temp dir that holds pocketic.port. There is no polling fallback and no deadline on that await (src/main.rs, rx.recv().await after the watcher is armed). If the FSEvents notification is missed, the port file sits there fully written and unread, and the launcher never proceeds.

Environment

• macOS 26.5 (25F71), Apple M3 Ultra, 96 GB
• icp CLI 1.2.0; launcher v15.0.0-2026-08-06, v15.0.0-2026-08-13 and v16.0.0 all reproduced it (the await code at src/main.rs HEAD is unchanged since those builds)
• A busy host: several worktrees each running their own local network (~13 pocket-ic servers / ~300 pocket-ic processes at the time), load average 15–18, fseventsd under sustained churn, ~1,100 entries under $TMPDIR (mostly other launchers' temp dirs)

Symptoms

Observed 2026-09-05, five consecutive starts in one worktree after a clean icp network stop, all identical (the previous day: one wedge, first retry worked):

• the CLI prints its two "output will be redirected" lines and never Network started on port;
• icp-cli-network-launcher is alive at 0% CPU with an empty --status-dir; sample shows every thread parked (__psynch_cvwait / kevent / an idle CFRunLoop helper — the notify FSEvents run loop). It holds no TCP connection to pocket-ic;
• the pocket-ic child is listening and answers GET /instances with [] (never got as far as instance creation), and its --port-file <tmp>/pocketic.port exists with the port and trailing newline in it;
• .icp/cache/networks/local/state is empty, the lock is held, no descriptor.json;
• touching or rewriting the port file after the wedge does not wake the launcher — which suggests the stream is not delivering to that process at all, not just one dropped event;
• ruled out: launcher version, env differences, DNS, port/lock holders, stray pocket-ic from earlier attempts, proc/file limits, proxy settings. A manual launcher run with RUST_LOG=debug prints only pocket-ic's "listening on port" line and then nothing.

Where it wedges

src/main.rs (port discovery):

let port_file = tmpdir.path().join("pocketic.port");
let (tx, mut rx) = tokio::sync::mpsc::channel(10);
let mut watcher = recommended_watcher({ /* on event: read port_file, send port */ })?;
watcher.watch(tmpdir.path(), RecursiveMode::Recursive)?;
// ... spawn pocket-ic with --port-file ...
let config_port = rx
    .recv()
    .await
    .expect("failed to receive port from watcher")?;

The only read of the port file happens inside the watcher callback, so port discovery is entirely at the mercy of FSEvents delivery. FSEvents is a best-effort, coalescing, daemon-mediated notification; under fseventsd pressure it can drop or fail to deliver, and there is nothing here to recover. Two smaller consequences of the same shape:

• an event that fires before the trailing \n lands is ignored (contents.ends_with('\n')), and the launcher then depends on a second event arriving;
• there is no timeout, so the failure mode is an indefinite hang with no message — the CLI's "Timed out waiting for PocketIC server" only covers the later client phase.

Suggested fix

Keep the watcher as the fast path, but do not depend on it:

1. Poll the port file (e.g. every 100–250 ms) as a fallback, racing the watcher channel — pocket-ic's own Rust client already discovers the server by polling its port file, so this matches upstream practice.
2. Put a deadline on the whole port-discovery step (e.g. 30–60 s, or the existing start timeout) and fail with a clear error naming the port file and the pocket-ic pid instead of parking.

Either of these alone would have turned a silent indefinite hang into a start that succeeds within a second (the file was there) or fails loudly.

Workaround

Kill the launcher and its pocket-ic child and retry; on a busy host this can take several retries (we wrote a small doctor script that detects the signature — launcher older than a grace period, empty status dir, port file present and pocket-ic answering [] — and does exactly that).

Start in src/main.rs at the port-discovery watcher and the rx.recv().await call, then inspect how pocket-ic writes the port file. Add a polling fallback and a deadline that reports the port file and pocket-ic process when discovery fails; done means a written port is discovered without watcher delivery and a missed file no longer hangs indefinitely.