cockroachdb / cockroachdb/cockroach
admission: compaction byte estimates can be too low
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
When there is almost no foreground work, and background work is bursty e.g. an import or index backfill, the compaction concurrency can vary significantly and be much lower than the configured max (in the following, the max was 3). In that case, using the observed compaction bytes out of L0 to model the capacity can be an under-estimate. We could scale this up by the (compaction-time-capacity-over-adjustment-interval)/(compaction-time-observed-over-adjustment-interval): the numerator in this example would be 3*15s = 45s. Note the AC throttline at 18:19:10 and 18:20:55 below (`admitting 33 MiB (rate 2.2 MiB/s) (elastic 20 MiB rate 1.3 MiB/s)` in the former case)
```
I260120 18:19:10.072730 873 util/admission/io_load_listener.go:787 ⋮ [T1,Vsystem,n1,s1] 2506 IO overload: compaction score 0.250 (10 ssts, 5 sub-levels), L0 growth 118 MiB (write 0 B (ignored 0 B) ingest 118 MiB (ignored 0 B)): requests 121 (0 bypassed) with 1.0 MiB acc-write (0 B bypassed) + 212 MiB acc-ingest (0 B bypassed) + 212 MiB adjusted-LSM-writes + 407 MiB adjusted-disk-writes + write-model 0.00x+0 B (smoothed 0.56x+1 B) + l0-ingest-model 0.55x+1 B (smoothed 0.34x+1 B) + ingest-model 1.00x+1 B (smoothed 1.00x+1 B) + write-amp-model 1.92x+1 B (smoothed 2.48x+1 B) + at-admission-tokens 8.8 KiB, compacted 27 MiB [≈23 MiB], flushed 0 B [≈0 B] (mult 1.00); admitting 33 MiB (rate 2.2 MiB/s) (elastic 20 MiB rate 1.3 MiB/s) due to L0 growth (used total: 28 MiB elastic 27 MiB); write stalls 0; diskBandwidthLimiter (unlimited) (writeUtil 0.00, tokensUsed (elastic 0 B, snapshot 0 B, regular 0 B) tokens (write 0 B (prev 0 B), read 0 B (prev 0 B)), writeBW 0 B/s, readBW 0 B/s, provisioned 0 B/s, err(cum,abs,acc) write: 0 B,0 B,0 B read: 0 B,0 B,0 B)
I260120 18:19:25.074040 873 util/admission/io_load_listener.go:787 ⋮ [T1,Vsystem,n1,s1] 2514 IO overload: compaction score 0.000 (0 ssts, 0 sub-levels), L0 growth 31 MiB (write 0 B (ignored 0 B) ingest 31 MiB (ignored 0 B)): requests 36 (0 bypassed) with 789 KiB acc-write (0 B bypassed) + 81 MiB acc-ingest (0 B bypassed) + 81 MiB adjusted-LSM-writes + 402 MiB adjusted-disk-writes + write-model 0.00x+0 B (smoothed 0.56x+1 B) + l0-ingest-model 0.39x+1 B (smoothed 0.36x+1 B) + ingest-model 1.00x+1 B (smoothed 1.00x+1 B) + write-amp-model 4.97x+1 B (smoothed 3.73x+1 B) + at-admission-tokens 14 KiB, compacted 122 MiB [≈72 MiB], flushed 0 B [≈0 B] (mult 1.00); admitting all (used total: 24 MiB elastic 23 MiB); write stalls 0; diskBandwidthLimiter (unlimited) (writeUtil 0.00, tokensUsed (elastic 0 B, snapshot 0 B, regular 0 B) tokens (write 0 B (prev 0 B), read 0 B (prev 0 B)), writeBW 0 B/s, readBW 0 B/s, provisioned 0 B/s, err(cum,abs,acc) write: 0 B,0 B,0 B read: 0 B,0 B,0 B)
I260120 18:20:55.076041 873 util/admission/io_load_listener.go:787 ⋮ [T1,Vsystem,n1,s1] 2541 IO overload: compaction score 0.450 (9 ssts, 9 sub-levels), L0 growth 125 MiB (write 0 B (ignored 0 B) ingest 125 MiB (ignored 0 B)): requests 76 (0 bypassed) with 791 KiB acc-write (0 B bypassed) + 208 MiB acc-ingest (0 B bypassed) + 208 MiB adjusted-LSM-writes + 850 MiB adjusted-disk-writes + write-model 0.00x+0 B (smoothed 1.43x+1 B) + l0-ingest-model 0.60x+1 B (smoothed 0.33x+1 B) + ingest-model 1.00x+1 B (smoothed 1.00x+1 B) + write-amp-model 4.07x+1 B (smoothed 4.84x+1 B) + at-admission-tokens 31 KiB, compacted 51 MiB [≈27 MiB], flushed 0 B [≈0 B] (mult 1.00); admitting 69 MiB (rate 4.6 MiB/s) (elastic 10 MiB rate 696 KiB/s) due to L0 growth(used token lower bound) (used total: 12 MiB elastic 11 MiB); write stalls 0; diskBandwidthLimiter (unlimited) (writeUtil 0.00, tokensUsed (elastic 0 B, snapshot 0 B, regular 0 B) tokens (write 0 B (prev 0 B), read 0 B (prev 0 B)), writeBW 0 B/s, readBW 0 B/s, provisioned 0 B/s, err(cum,abs,acc) write: 0 B,0 B,0 B read: 0 B,0 B,0 B)
```
Jira issue: CRDB-58877
Contributor guide
Assessment
This issue has not been assessed yet.