Improve resolution and value of collected stats...

A few suggestions:

1) There should probably be three metric intervals: high, medium, and low frequency metrics. High frequency metrics can be collected as frequently as once every 100ms (e.g. CPU, Go memory usage, etc). Medium metrics once every 10s (I'm not sure what would need to be polled that infrequently), and low frequency metrics every ~60s or 300s (e.g. disk usage, etc). It's been my experience that decoupling CPU from disk stats is important when the frequency is sub-second because of the number of locks acquired by the VFS to report out stats (whereas CPU counters are much cheaper).

2) All high and medium-frequency metrics of Gauge type should be converted from `SetGauge()` to `AddSample()`.

3) The conditional guarding the per-allocation metrics should remain in place. Maybe a new nob, "collect high-frequency stats" that, when enabled, uses the high-frequency timeout loop, or when disabled, uses the medium-frequency timeout.

https://github.com/hashicorp/nomad/blob/8b6fcd34830a661951920070aaccc66d3377c4b0/client/client.go#L2092-L2116

Start in client/client.go around lines 2092-2116 and trace how collected metrics are scheduled and recorded. Compare the existing allocation-metrics condition and gauge handling with the proposed high-, medium-, and low-frequency intervals. Done means the collection behavior, metric types, and optional high-frequency setting match the agreed design, with relevant tests updated or added.

Written by the indexing model from the issue text.