kvserver: always-on latency histograms at leader for per-replica persistence of raft log entry

https://github.com/cockroachdb/cockroach/pull/131850 added tracing for Raft log entries. But tracing is usually disabled by default. We see escalations where a cluster has suffered a one time high latency event, sometimes lasting only a couple of minutes. Diagnosing the root cause for these events is important, but is hard, especially in large clusters, since one does not know which nodes/stores were responsible for slow writes.

We already track per-entry state efficiently at the leader in replicaton admission control v2, for token return. So it is feasible to have an always-on mode that tracks for each entry the latency that was observed at the leader, for each replica, in persisting that entry (for the local replica via MsgStorageAppendResp, and for a remote replica via MsgAppResp). This could be added to per store latency histograms. Exporting such latency histograms as metrics is not viable, since it requires each node to export a histogram for every other store in the system. We already have a pattern for exporting such state via logs in replication AC which we could reuse here: periodically iterate over the histograms (and reset them, so these are deltas); if the 99th percentile latency for a store is above some threshold and in the top-k across stores, include it in a log statement (with a few percentiles printed for each store).

These always-on histograms can quickly pinpoint which node/store to investigate further, and removes guesswork.

If we want to be more ambitious we could also have a histogram per store that includes only the latency values when it was the last store to contribute to quorum being achieved. This filters out stores that were slow but not responsible for the high latency anomaly.

Jira issue: CRDB-43613

Epic CRDB-42900