Track resource use per zone

The other day, I was thinking about how a given oximeter workload might affect the resource use of the clickhouse and nexus services. I didn't see a great way to do this: we run multiple nexus instances, and can run multiple clickhouse instances, and I didn't want to ssh into each relevant zone and tail zonestat for each one. I slapped together a MacGyver-style resource use exporter to gather the data I wanted:

• On each sled, add a crontab entry that writes per-zone usage to a file:

* * * * * zonestat -p 10 6 >> /var/log/zonestat-$(/usr/bin/date +\%Y\%m\%d)

• Run a local VictoriaMetrics instance:

docker run -d --name victoriametrics -p 8428:8428 victoriametrics/victoria-metrics

• Collect the zonestat files from the sleds, then ingest into VictoriaMetrics with a vibe-coded script:

for sled in $(07 10 14 21); scp gc$sled:/var/log/zonestat "./zonestat/sled-$sled"; end
for path in $(ls zonestat/*/zonestat-*); python3 parse_zonestat.py $path | curl -X POST 'http://localhost:8428/api/v1/import' --data-binary @-; end

Now we have a promql-compatible service we can use to query basic resource use (cpu, physical memory, virtual memory) over all zones:

This sort of works, even though the "implementation" is a one-line cron entry and a python script. However, the cron entries are purged when the sleds are updated, and rsyncing text files isn't a great interface for fetching the data. I'm hoping we can build a proper version of this feature, so that we can track per-zone resource use without manually slinging files around. A few options:

• Maybe the most obvious approach would be to add zone-level resource use metrics to oximeter via sled-agent.
• We could also put these stats somewhere else if we don't want them to be visible to customers. Should customers care about the resource use of nexus or clickhouse? Should they even know that those services exist? If not, we could dump zone stats to a different table in clickhouse that's not exposed to customers. @smklein may have suggested this last Demo Friday.

Note: I used zonestat for this popsickle-sticks-and-glue implementation for simplicity, but we might want to use kstat instead if we decide to make this a real feature. But note that kstat, unlike zonestat, can't track zone memory use for zones without a memory cap, and we don't set memory caps at the moment. Gemini thinks that zonestat falls back to rusagesys in this case. So we could continue wrapping zonestat, or teach sled-agent to wrap rusagesys, or introduce resource caps (possibly making them practically infinite). Or maybe, per @davepacheco's post, we should look at more granular memory pressure stats instead or in addition to utilization.

In the meantime, we can fall back to the zonestat / VictoriaMetrics version if we want to know about zone resource use.

cc @bnaecker

Start by reading sled-agent and the existing Oximeter metric path, then compare the zonestat, kstat, and rusagesys approaches described here. Resolve where zone-level CPU and memory metrics should be stored and whether they are customer-visible; done means resource use can be queried without cron entries, SSH, or manually copied files.