allocator: high amount of CPU usage for storepool logging

In an internal test cluster running 23.1.2 with 48 nodes we see that allocator logging is using 4% of CPU (32vCPU).


The CPU usage is from here:

https://github.com/cockroachdb/cockroach/blob/fdf04ffe5403daab580ab0a6fb0ad407ccc23c2e/pkg/kv/kvserver/allocator/storepool/store_pool.go#L1421-L1427

When grabbing another profile the CPU time disappears. The suspect part is the number of string operations - which eat the majority of CPU shown above.

Details


It appears possible redact printing `status` is eating most of the CPU, however I'm not certain. Also these log statements are at `v=3`, meaning that tracing must have been enabled consistently - which I suppose makes sense given:

https://github.com/cockroachdb/cockroach/blob/fdf04ffe5403daab580ab0a6fb0ad407ccc23c2e/pkg/kv/kvserver/replicate_queue.go#L721-L723

Jira issue: CRDB-28774