kvcoord: DistSender concurrency causing goroutine scheduling latency spikes

When DistSender receives a batch that spans multiple ranges, it can send partial batches asynchronously and is governed by `kv.dist_sender.concurrency_limit`, which defaults to `DefaultSenderStreamsPerVCPU*max(MinViableProcs, runtime.GOMAXPROCS(0))` (i.e. **384 async sends per vCPU**, with a minimum of **2 vCPUs**).

Code reference: https://github.com/cockroachdb/cockroach/blob/f395f6aa8728cc7d97b906f17b38af7f7c783577/pkg/kv/kvclient/kvcoord/dist_sender.go#L386-L419

This default was chosen based on its ability to more fully utilize CPUs in certain workloads; it was increased (6x) in https://github.com/cockroachdb/cockroach/pull/131226

We’ve seen in escalations[^1] that this can drive up goroutine scheduling latency (to the point where elastic traffic is always throttled as much as possible).

We should devise a mechanism that more intelligently allows parallelism when capacity is available, but avoids oversubscribing the scheduler as seen here. Intuitively, a couple of goroutines are “foreground traffic”, but once you have a few, maybe additional ones should count as “elastic” and their creation could be deferred intelligently.

This issue mirrors https://github.com/cockroachdb/cockroach/issues/129942, so this problem needs to be solved in at least two places, but possibly additional ones exist (across the entire codebase, not just KV).

[^1]: https://github.com/cockroachlabs/support/issues/3483

Jira issue: CRDB-59414