cockroachdb / cockroachdb/cockroach

admission: use recent history of sub-level count when sub-level count is below 5

Open
#109,361 0 comments 0 reactions 0 assignees View on GitHub
A-admission-control C-enhancement
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Despite persistent overload and increasing compaction debt, L0 sub-level count can occasionally fall below 5.

For example:
```
I230823 18:06:08.874355 448 util/admission/io_load_listener.go:544 ⋮ [T1,Vsystem,n1,s1] 2594 IO overload: compaction score 0.500 (262 ssts, 10 sub-levels), L0 growth 742 MiB (write 742 MiB ingest 0 B ignored 0 B): requests 12594 (0 bypassed) with 390 MiB acc-write (0 B bypassed) + 0 B acc-ingest (0 B bypassed) + write-model 1.90x+1 B (smoothed 1.84x+1 B) + ingested-model 0.00x+0 B (smoothed 0.75x+1 B) + at-admission-tokens 59 KiB, compacted 644 MiB [≈688 MiB], flushed 1.6 GiB [≈1.6 GiB]; admitting 691 MiB (rate 46 MiB/s) due to L0 growth (used 694 MiB); write stalls 0

I230823 18:06:23.874741 448 util/admission/io_load_listener.go:544 ⋮ [T1,Vsystem,n1,s1] 2595 IO overload: compaction score 0.200 (65 ssts, 4 sub-levels), L0 growth 624 MiB (write 624 MiB ingest 0 B ignored 0 B): requests 11927 (0 bypassed) with 373 MiB acc-write (0 B bypassed) + 0 B acc-ingest (0 B bypassed) + write-model 1.67x+1 B (smoothed 1.76x+1 B) + ingested-model 0.00x+0 B (smoothed 0.75x+1 B) + at-admission-tokens 56 KiB, compacted 976 MiB [≈832 MiB], flushed 1.6 GiB [≈1.6 GiB]; admitting 2.2 GiB (rate 151 MiB/s) due to memtable flush (multiplier 1.375) (used 691 MiB); write stalls 0

<13 write stalls; used 1.3GiB is what we admitted>
I230823 18:06:38.875720 448 util/admission/io_load_listener.go:544 ⋮ [T1,Vsystem,n1,s1] 2970 IO overload: compaction score 0.600 (424 ssts, 12 sub-levels), L0 growth 1.3 GiB (write 1.3 GiB ingest 0 B ignored 0 B): requests 15006 (0 bypassed) with 770 MiB acc-write (0 B bypassed) + 0 B acc-ingest (0 B bypassed) + write-model 1.68x+1 B (smoothed 1.72x+1 B) + ingested-model 0.00x+0 B (smoothed 0.75x+1 B) + at-admission-tokens 72 KiB, compacted 589 MiB [≈711 MiB], flushed 1.4 GiB [≈1.5 GiB]; admitting 736 MiB (rate 49 MiB/s) due to L0 growth (used 1.3 GiB); write stalls 13

I230823 18:06:53.876084 448 util/admission/io_load_listener.go:544 ⋮ [T1,Vsystem,n1,s1] 3048 IO overload: compaction score 0.750 (444 ssts, 15 sub-levels), L0 growth 1002 MiB (write 1002 MiB ingest 0 B ignored 0 B): requests 14570 (0 bypassed) with 442 MiB acc-write (0 B bypassed) + 0 B acc-ingest (0 B bypassed) + write-model 2.26x+1 B (smoothed 1.99x+1 B) + ingested-model 0.00x+0 B (smoothed 0.75x+1 B) + at-admission-tokens 71 KiB, compacted 1.1 GiB [≈908 MiB], flushed 1.6 GiB [≈1.5 GiB]; admitting 709 MiB (rate 47 MiB/s) due to L0 growth (used 739 MiB); write stalls 0

I230823 18:07:08.877297 448 util/admission/io_load_listener.go:544 ⋮ [T1,Vsystem,n1,s1] 3049 IO overload: compaction score 0.400 (178 ssts, 8 sub-levels), L0 growth 681 MiB (write 681 MiB ingest 0 B ignored 0 B): requests 11770 (0 bypassed) with 368 MiB acc-write (0 B bypassed) + 0 B acc-ingest (0 B bypassed) + write-model 1.85x+1 B (smoothed 1.92x+1 B) + ingested-model 0.00x+0 B (smoothed 0.75x+1 B) + at-admission-tokens 65 KiB, compacted 1.1 GiB [≈1021 MiB], flushed 1.6 GiB [≈1.6 GiB]; admitting 1.0 GiB (rate 71 MiB/s) due to L0 growth (used 709 MiB); write stalls 0

I230823 18:07:23.877584 448 util/admission/io_load_listener.go:544 ⋮ [T1,Vsystem,n1,s1] 3052 IO overload: compaction score 0.700 (399 ssts, 14 sub-levels), L0 growth 968 MiB (write 968 MiB ingest 0 B ignored 0 B): requests 17186 (0 bypassed) with 546 MiB acc-write (0 B bypassed) + 0 B acc-ingest (0 B bypassed) + write-model 1.77x+1 B (smoothed 1.85x+1 B) + ingested-model 0.00x+0 B (smoothed 0.75x+1 B) + at-admission-tokens 62 KiB, compacted 631 MiB [≈826 MiB], flushed 1.5 GiB [≈1.5 GiB]; admitting 865 MiB (rate 58 MiB/s) due to L0 growth (used 1.0 GiB); write stalls 0

I230823 18:07:38.878178 448 util/admission/io_load_listener.go:544 ⋮ [T1,Vsystem,n1,s1] 3053 IO overload: compaction score 0.600 (299 ssts, 12 sub-levels), L0 growth 855 MiB (write 855 MiB ingest 0 B ignored 0 B): requests 14933 (0 bypassed) with 463 MiB acc-write (0 B bypassed) + 0 B acc-ingest (0 B bypassed) + write-model 1.85x+1 B (smoothed 1.85x+1 B) + ingested-model 0.00x+0 B (smoothed 0.75x+1 B) + at-admission-tokens 60 KiB, compacted 904 MiB [≈865 MiB], flushed 1.5 GiB [≈1.5 GiB]; admitting 821 MiB (rate 55 MiB/s) due to L0 growth (used 865 MiB); write stalls 0

I230823 18:07:53.878616 448 util/admission/io_load_listener.go:544 ⋮ [T1,Vsystem,n1,s1] 3055 IO overload: compaction score 0.900 (407 ssts, 18 sub-levels), L0 growth 798 MiB (write 798 MiB ingest 0 B ignored 0 B): requests 14191 (0 bypassed) with 444 MiB acc-write (0 B bypassed) + 0 B acc-ingest (0 B bypassed) + write-model 1.80x+1 B (smoothed 1.82x+1 B) + ingested-model 0.00x+0 B (smoothed 0.75x+1 B) + at-admission-tokens 59 KiB, compacted 648 MiB [≈756 MiB], flushed 1.5 GiB [≈1.5 GiB]; admitting 638 MiB (rate 42 MiB/s) due to L0 growth (used 821 MiB); write stalls 0
```

See https://docs.google.com/document/d/1yqh2j3l6kjDVIkI5qxTH8xVTx9e2_pcFFJ7QW_JDgi0/edit#heading=h.tul27fvgwgg2 for details.

This can be improved by realizing that the sub-levels have recently dropped below 5, and more gradually increasing the tokens.

@bananabrick

Jira issue: CRDB-30889

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.