[audit][perf] Global atomic stat counters updated on every read/write syscall bounce shared cache lines across all net threads

Severity: medium · Category: performance
Location: src/iocore/net/UnixNetVConnection.cc:542

What's wrong

The metrics subsystem stores one global std::atomic<int64_t> per stat, packed adjacently in spans (include/tsutil/Metrics.h:64-68, fetch_add with memory_order_relaxed), with no per-thread or per-core sharding and no cache-line alignment. The net fast path increments net_rsb.calls_to_read on every recvmsg (line 542), read_bytes and read_bytes_count on every successful read (lines 574-575), and calls_to_write on every sendmsg iteration (line 882) — several RMWs per IO event, all threads targeting the same few cache lines, with adjacent unrelated counters false-sharing the same 64B lines. At high event rates this is a measurable cross-core coherence cost per syscall, not per request. The same pattern appears per request in remap: url_mapping::_hitCount is a std::atomic<uint64_t> incremented with default seq_cst ordering on every matched request (include/proxy/http/remap/UrlMapping.h:120,169), a single hot line when one rule dominates traffic.

Evidence

UnixNetVConnection.cc:542: Metrics::Counter::increment(net_rsb.calls_to_read);
UnixNetVConnection.cc:574-575: Metrics::Counter::increment(net_rsb.read_bytes, r); Metrics::Counter::increment(net_rsb.read_bytes_count);
Metrics.h:65-68: increment(int64_t val) { _value.fetch_add(val, MEMORY_ORDER); }
UrlMapping.h:167-170: incrementCount() { _hitCount++; }

Suggested fix

Shard hot counters per event thread (each thread updates its own padded slot; readers sum on scrape, as the old RecRaw stats did), or at minimum batch per-IO counters into thread-locals flushed periodically; make _hitCount use fetch_add(1, std::memory_order_relaxed).

Filed from an automated multi-lens codebase audit. Full report: CODEBASE_AUDIT.md / audit-report.html on branch claude/codebase-audit-review-9nw7vz.

Start with include/tsutil/Metrics.h and the cited read/write paths in src/iocore/net/UnixNetVConnection.cc, then inspect include/proxy/http/remap/UrlMapping.h. Compare the existing global and per-rule atomic updates with the old RecRaw stats approach mentioned in the issue. Done means reducing hot-path cross-thread contention through sharding or batching and using relaxed ordering for _hitCount, with the resulting behavior and metric reads verified.