admission: write-amp model fluctuates significantly

See the sequence of interval write-amp models, where the * represent WAL failover:
14.08x+1 B
9.03x+1 B*
60.96x+1 B*
73.13x+1 B
43.00x+1 B
16.57x+1 B

In theory, during a write stall that triggers a WAL failover, the write-amp model will incorrectly have a reduced coefficient, since compactions may not be happening. So there is something to improve there. But in this example, it was a latency induced WAL failover where the primary was able to do compactions, so we see write-amp both decrease and increase during WAL failover. And it is high even after WAL failover. It is unclear what to do here.

This was observed in the test discussed in https://cockroachlabs.slack.com/archives/C06UFBJ743F/p1746549322239219?thread_ts=1746051766.185039&cid=C06UFBJ743F

Detailed logs:
I250506 16:49:51.649298 932 util/admission/io_load_listener.go:665 ⋮ [T1,Vsystem,n5,s19] 5965 IO overload: compaction score 0.100 (50 ssts, 2 sub-levels), L0 growth 64 MiB (write 40 MiB (ignored 0 B) ingest 23 MiB (ignored 0 B)): requests 3988 (1940 bypassed) with 90 MiB acc-write (541 KiB bypassed) + 26 MiB acc-ingest (0 B bypassed) + 66 MiB adjusted-LSM-writes + 931 MiB adjusted-disk-writes + write-model 0.50x+1 B (smoothed 0.50x+1 B) + ingested-model 0.90x+1 B (smoothed 0.90x+1 B) + write-amp-model 14.08x+1 B (smoothed 13.47x+1 B) + at-admission-tokens 1.9 KiB, compacted 61 MiB [≈64 MiB], flushed 485 MiB [≈495 MiB] (mult 1.00); admitting elastic 79 MiB (rate 5.3 MiB/s) due to L0 growth; write stalls 0; diskBandwidthLimiter (tokenUtilization 0.71, tokensUsed (elastic 942 MiB, snapshot 0 B, regular 56 MiB) tokens (write 1.4 GiB (prev 1.4 GiB)), writeBW 62 MiB/s, readBW 7.4 MiB/s, provisioned 125 MiB/s)

I250506 16:50:06.649916 932 util/admission/io_load_listener.go:665 ⋮ [T1,Vsystem,n5,s19] 5991 IO overload: compaction score 0.200 (52 ssts, 4 sub-levels), L0 growth 79 MiB (write 49 MiB (ignored 0 B) ingest 29 MiB (ignored 0 B)): requests 4315 (1700 bypassed) with 108 MiB acc-write (462 KiB bypassed) + 33 MiB acc-ingest (0 B bypassed) + 82 MiB adjusted-LSM-writes + 740 MiB adjusted-disk-writes + write-model 0.50x+1 B (smoothed 0.50x+1 B) + ingested-model 0.90x+1 B (smoothed 0.90x+1 B) + write-amp-model 9.03x+1 B (smoothed 11.25x+1 B) + at-admission-tokens 1.7 KiB, compacted 74 MiB [≈64 MiB], flushed 192 MiB [≈495 MiB] (mult 1.00); admitting (WAL failover) elastic 1 B (rate 0 B/s) due to L0 growth; write stalls 0; diskBandwidthLimiter (tokenUtilization 0.81, tokensUsed (elastic 1.0 GiB, snapshot 0 B, regular 59 MiB) tokens (write 538 MiB (prev 1.4 GiB)), writeBW 49 MiB/s, readBW 64 MiB/s, provisioned 125 MiB/s)

I250506 16:50:21.650490 932 util/admission/io_load_listener.go:665 ⋮ [T1,Vsystem,n5,s19] 6010 IO overload: compaction score 0.050 (10 ssts, 1 sub-levels), L0 growth 12 MiB (write 10 MiB (ignored 0 B) ingest 2.3 MiB (ignored 0 B)): requests 3644 (1387 bypassed) with 16 MiB acc-write (386 KiB bypassed) + 2.3 MiB acc-ingest (0 B bypassed) + 12 MiB adjusted-LSM-writes + 757 MiB adjusted-disk-writes + write-model 0.64x+1 B (smoothed 0.57x+1 B) + ingested-model 1.00x+1 B (smoothed 0.95x+1 B) + write-amp-model 60.96x+1 B (smoothed 36.10x+1 B) + at-admission-tokens 2.0 KiB, compacted 44 MiB [≈64 MiB], flushed 406 MiB [≈495 MiB] (mult 1.00); admitting (WAL failover) all; write stalls 0; diskBandwidthLimiter (tokenUtilization 0.10, tokensUsed (elastic 5.6 MiB, snapshot 0 B, regular 48 MiB) tokens (write 967 MiB (prev 538 MiB)), writeBW 50 MiB/s, readBW 6.9 MiB/s, provisioned 125 MiB/s)

I250506 16:50:36.650972 932 util/admission/io_load_listener.go:665 ⋮ [T1,Vsystem,n5,s19] 6018 IO overload: compaction score 0.050 (12 ssts, 1 sub-levels), L0 growth 8.9 MiB (write 4.0 MiB (ignored 0 B) ingest 4.9 MiB (ignored 0 B)): requests 3306 (1397 bypassed) with 24 MiB acc-write (397 KiB bypassed) + 7.0 MiB acc-ingest (0 B bypassed) + 11 MiB adjusted-LSM-writes + 804 MiB adjusted-disk-writes + write-model 0.50x+1 B (smoothed 0.53x+1 B) + ingested-model 0.69x+1 B (smoothed 0.82x+1 B) + write-amp-model 73.13x+1 B (smoothed 54.62x+1 B) + at-admission-tokens 1.6 KiB, compacted 6.6 MiB [≈35 MiB], flushed 371 MiB [≈495 MiB] (mult 1.00); admitting all; write stalls 0; diskBandwidthLimiter (tokenUtilization 1.04, tokensUsed (elastic 836 MiB, snapshot 0 B, regular 174 MiB) tokens (write 1.4 GiB (prev 967 MiB)), writeBW 54 MiB/s, readBW 7.7 MiB/s, provisioned 125 MiB/s)

I250506 16:50:51.651465 932 util/admission/io_load_listener.go:665 ⋮ [T1,Vsystem,n5,s19] 6029 IO overload: compaction score 0.050 (15 ssts, 1 sub-levels), L0 growth 18 MiB (write 14 MiB (ignored 0 B) ingest 4.1 MiB (ignored 0 B)): requests 3055 (1051 bypassed) with 39 MiB acc-write (297 KiB bypassed) + 4.1 MiB acc-ingest (0 B bypassed) + 18 MiB adjusted-LSM-writes + 771 MiB adjusted-disk-writes + write-model 0.50x+1 B (smoothed 0.52x+1 B) + ingested-model 1.00x+1 B (smoothed 0.91x+1 B) + write-amp-model 43.00x+1 B (smoothed 48.81x+1 B) + at-admission-tokens 1.6 KiB, compacted 14 MiB [≈25 MiB], flushed 457 MiB [≈495 MiB] (mult 1.00); admitting all; write stalls 0; diskBandwidthLimiter (tokenUtilization 1.00, tokensUsed (elastic 1.1 GiB, snapshot 0 B, regular 245 MiB) tokens (write 1.4 GiB (prev 1.4 GiB)), writeBW 51 MiB/s, readBW 1.6 MiB/s, provisioned 125 MiB/s)

I250506 16:51:06.651920 932 util/admission/io_load_listener.go:665 ⋮ [T1,Vsystem,n5,s19] 6035 IO overload: compaction score 0.050 (5 ssts, 1 sub-levels), L0 growth 27 MiB (write 20 MiB (ignored 0 B) ingest 6.7 MiB (ignored 0 B)): requests 3207 (953 bypassed) with 48 MiB acc-write (267 KiB bypassed) + 12 MiB acc-ingest (0 B bypassed) + 32 MiB adjusted-LSM-writes + 528 MiB adjusted-disk-writes + write-model 0.50x+1 B (smoothed 0.51x+1 B) + ingested-model 0.57x+1 B (smoothed 0.74x+1 B) + write-amp-model 16.57x+1 B (smoothed 32.69x+1 B) + at-admission-tokens 1.6 KiB, compacted 35 MiB [≈30 MiB], flushed 501 MiB [≈495 MiB] (mult 1.00); admitting all; write stalls 0; diskBandwidthLimiter (tokenUtilization 0.98, tokensUsed (elastic 1.2 GiB, snapshot 0 B, regular 213 MiB) tokens (write 1.4 GiB (prev 1.4 GiB)), writeBW 35 MiB/s, readBW 4.4 MiB/s, provisioned 125 MiB/s)

I250506 16:51:21.652371 932 util/admission/io_load_listener.go:665 ⋮ [T1,Vsystem,n5,s19] 6049 IO overload: compaction score 0.050 (11 ssts, 1 sub-levels), L0 growth 32 MiB (write 24 MiB (ignored 0 B) ingest 8.4 MiB (ignored 0 B)): requests 3164 (810 bypassed) with 56 MiB acc-write (224 KiB bypassed) + 13 MiB acc-ingest (0 B bypassed) + 37 MiB adjusted-LSM-writes + 467 MiB adjusted-disk-writes + write-model 0.50x+1 B (smoothed 0.50x+1 B) + ingested-model 0.63x+1 B (smoothed 0.68x+1 B) + write-amp-model 12.69x+1 B (smoothed 22.69x+1 B) + at-admission-tokens 1.6 KiB, compacted 29 MiB [≈29 MiB], flushed 481 MiB [≈495 MiB] (mult 1.00); admitting all; write stalls 0; diskBandwidthLimiter (tokenUtilization 1.00, tokensUsed (elastic 1.3 GiB, snapshot 0 B, regular 140 MiB) tokens (write 1.4 GiB (prev 1.4 GiB)), writeBW 31 MiB/s, readBW 2.4 MiB/s, provisioned 125 MiB/s)

Jira issue: CRDB-50435