gateway: performance findings from a hot-path sweep

Start with .agent/CODING_TASTE.md and main_service.rs:2321, then trace the listed call sites and related proxy paths. Use the referenced module documentation and, where needed, the bench rig to establish a scoped change and baseline; done means the selected hotspot is addressed without changing correctness and its performance claim is measured.

A read of the gateway and certbot hot paths (proxy.rs, proxy/*, main_service.rs, kv/, certbot/src/), looking specifically for cost rather than correctness. The correctness findings from the same audit are already in PRs; this is what is left that is slow rather than wrong.

Ranked by cost × frequency. Each item says what it costs and roughly what the fix is.

The big one: blocking work under the routing mutex

1. wg syncconf (fork+exec) and a file write run while holding ProxyState — main_service.rs:2321 (reconfigure_inner), called from :725, :1611, :1338, :2562.

ProxyState is a std::sync::Mutex taken on the data path for every proxied connection. reconfigure_inner renders the config, does a synchronous file write, then cmd!(wg syncconf …) — a blocking fork/exec plus the child's netlink work — all inside the guard, and three of the four call sites are on a tokio worker with no spawn_blocking. For the duration (single-digit to low tens of ms) every proxy worker that needs to route is parked.

It fires once per CVM registration, once per remote instance change delivered by sync, and once per node change. CVMs re-register every ~3 minutes, so a fleet of N instances is roughly N/180 stalls per second, each freezing all routing.

.agent/CODING_TASTE.md records this exact rule from #740 — "no blocking commands or syscalls under a lock (wg show under ProxyState)". wg show was fixed by LatestHandshakesCache; wg syncconf was not.

Fix: render under the lock into an owned String, drop the guard, write + syncconf outside it (serialised by a separate apply mutex), and spawn_blocking at the async call sites. ~40 lines.

Three more places hold the same lock across work that does not need it:

where	held across	when
main_service.rs:1501-1612 (reload)	2 KV reads + 2 key format!s per instance, N KV writes, one InstanceInfo clone per instance, then the wg apply	every replicated instance/override change
admin_service.rs:55-76 (status)	wg show on a cold cache, N KV writes, N prefix scans	every dashboard poll. ProxyInner::latest_handshakes exists with a doc comment explaining why not to do this, and health_check::select_targets obeys it
main_service.rs:1452-1461 (connection sync)	N format!s + N msgpack encodes + N ephemeral write-lock acquisitions	every 30 s by default

admin_service.rs:282-311 (get_global_connections) has the identical shape and should move in the same change.

Per-connection cost on the proxy data path

2. Five separate acquisitions of the global routing mutex per connection — tls_terminate.rs:366-370, port_policy.rs:67 and :137-144, tls_passthough.rs:287-288.

select_top_n_hosts takes it once; is_port_allowed takes it once per candidate address (3 at the default connect_top_n); should_send_pp takes it again. The thread-per-core design in proxy.rs exists to remove exactly this kind of cross-thread coordination — its module doc measures 0.6 context switches per request as worth eliminating — and then the routing lookup reintroduces a process-global serialisation point five times per connection. It is also the same lock item 1 holds across a subprocess, so the two compound.

Fix: one select_routable(app_id, port) that filters by port policy and records the PP flag while the guard is already held. ~60 lines.

3. select_top_n_hosts allocates 3 Strings per connection out of the cache that exists to avoid work — main_service.rs:2382, proxy.rs:40.

The cached fast path is return Ok(top_n.clone()), and each AddressInfo owns a String instance_id. So a cache hit costs 3 mallocs + 3 frees per connection. At the 245k conn/s the balance.rs module doc quotes, that is ~735k malloc/free pairs per second copying an id that is immutable for the life of the record.

Fix: Arc<str>. The downstream consumers already take &str. ~15 lines.

4. proxy_to_app clones the whole address group to build an error message that is almost never produced — tls_passthough.rs:289-296. Doubles item 3's cost on the passthrough path. The sibling terminate path does it correctly (tls_terminate.rs:497 moves the group and names only the app), so this is an asymmetry rather than a taste call.

5. adaptive_ktls::relay_until allocates 64 KiB of zeroed buffer per connection — adaptive_ktls.rs:56-57. Sixty lines away, the structurally identical splice::relay_until uses a thread-local PooledBufs with a comment saying "a pair of buffer_size allocations per connection is a real cost when the connection only carries a few hundred bytes". It also ignores both config.buffer_size and the RELAY_BUF_SIZE constant declared for the purpose. Fix: make PooledBufs pub(super) and use it. ~6 lines.

6. latest_handshakes() rebuilds and re-clones the whole peer map on every call — handshakes.rs:127-153. add_elapsed_time builds a fresh BTreeMap cloning every 44-byte base64 pubkey. Callers include random_select_a_host, which the module doc at main_service.rs:1660 says "has no cache and runs per connection". A 500-peer fleet pays 500 allocations per call. Fix: return the Arc the cell already stores plus an age() helper. ~40 lines over five call sites.

7. PROXY-protocol v1 header is read one byte per syscall — pp.rs:160-176. 40–100 read(2) calls plus as many future polls per connection, where one read would do. Only paid with inbound_pp_enabled = true, which is the configuration for anyone behind HAProxy or a v1-speaking L4 LB. ~50–100 µs added to a setup the splice module measures at ~134 µs total. Fix: one bounded read, find \r\n, return the remainder so take_sni can seed its buffer with it instead of allocating a fresh 4 KiB. ~30 lines.

8. Balancer::target() scans all 32 counters on shared cache lines per accepted connection — balance.rs:124-142. counts: Arc<Vec<AtomicUsize>> packs 32 counters into 4 cache lines shared by all 32 cores, each doing a read-modify-write per connection open and close. The module doc cites HAProxy's multi-queue accept, which picks the least loaded of three candidates — the citation is there, the bounded sample is not.

I cannot say how slow this is without a run on the bench rig; the effect is invisible at the measured 16 connections over 4 cores and grows with worker count. The fix is cheap (pad to a cache line, sample self + 2 random, ~25 lines) and the shape is known-bad, but it should be re-measured against the numbers already in the module doc before landing.

Repeated work

9. Reload decodes every inst/ record three times, twice for a result it throws away — main_service.rs:1482, :1490; kv/mod.rs:1110, :1141. migrate_legacy_instance_overrides() internally runs legacy_instance_overrides(), then the caller runs it again, then load_all_instances() decodes the same bytes a third time. In a single-version cluster both legacy scans return empty — pure overhead, forever, not just during an upgrade. Fix: return the map the migration already computed. ~10 lines.

10. CloudflareClient builds a fresh reqwest::Client for every API call — dns01_client/cloudflare.rs:89, :167, :191, :217. Each Client::new() builds a connection pool and a TLS config including loading the root certificate store, so every DNS-01 record add and delete pays a root-store build plus a full TLS handshake, and connection reuse is impossible. .agent/CODING_TASTE.md names this rule from #741. certbot/src/http_client.rs already does it correctly for the ACME transport — same crate, opposite habit. Fix: hold the client on the struct. ~15 lines.

11. resolve_zone_id lists every zone in the Cloudflare account on every client construction — dns01_client/cloudflare.rs:82-163. Pages GET /zones 50 at a time and builds a map of the whole account before writing a single record; called per domain per renewal. The API supports GET /zones?name=<candidate>, which would be 2–3 requests instead of ceil(zones/50) — at the cost of one extra round trip for a single-zone account. Worth a decision rather than a patch.

12. Dead exact_certs map costs a hash of the SNI on every TLS handshake — cert_store.rs:36-38, :57-62, :229, :270. Documented as "Exact domain -> CertifiedKey" and probed first in resolve_cert, but nothing ever inserts into it. The wasted hash is small; the bigger cost is that a reader debugging "why isn't my exact-domain cert served" finds code that looks like it should work. Fix: delete it. ~12 lines removed.

Needs a decision rather than a patch

13. The sync endpoint decompresses up to 128 MiB and merges the whole store on an async worker — web_routes/wavekv_sync.rs:151-182, :206-235. Does not block the proxy (separate runtime) but does block the pool that serves registration and the admin dashboard. The MAX_DECOMPRESSED_SYNC_BYTES doc at kv/mod.rs:638 explicitly reasons about a peer running a buggy build sending a 128 MiB expansion; that peer also gets to pin a worker for as long as inflating and merging it takes. spawn_blocking is easy in principle but means moving the &Proxy borrow across the runtime boundary. Whether to engineer for the ceiling or the typical case (max_delta_bytes defaults to 4 MiB, three orders below) is a maintainer call.

14. take_sni re-parses the whole buffer after every read, and reports "no SNI" for a ClientHello over 4 KiB — proxy.rs:90-112. The re-parse is O(k·n) in segments, theoretical at k=1. The 4 KiB cap is real: once data_len == 4096 the read lands in an empty slice and returns Ok(0), which reads as EOF, so an oversized ClientHello fails as "no sni found" rather than "handshake too large". Post-quantum key shares (X25519MLKEM768 ≈ 1.2 KiB) still fit; a client stacking several groups plus a large session ticket will not. Grow on demand to a stated cap, or keep 4 KiB and make the overflow an explicit error naming the limit — a policy choice. Best folded into item 7, which already touches this buffer.

Checked and found clean

So a later reader knows where not to look again: proxy/splice.rs (the pipe pool, the try_io-before-readable path with its measured 2.4% justification, the drained invariant, the kTLS EINVAL handling, release_idle_pipes descriptor accounting), proxy/io_bridge.rs, proxy/idle.rs, proxy/sni.rs, proxy/health_check.rs (set-membership retain with an explicit note about the quadratic form it replaced, cursor rotation, bounded and sanitised log reasons, measured round budget), the cert_store.rs read path, proxy/reuseport.rs, proxy/stats.rs, kv/compat.rs, kv/sync_service.rs, kv/https_client.rs, metrics.rs and web_routes/metrics.rs, certbot/src/acme_client.rs, certbot/src/http_client.rs.

Several of these carry measurements in their comments and the claims hold up; the hot path is in good shape apart from the lock scope.