cockroachdb / cockroachdb/cockroach
admission: improve modeling of write bytes to flushed bytes
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Motivated by https://github.com/cockroachlabs/support/issues/3470 and see https://docs.google.com/document/d/19myJPjd9lxbhPu-4KYTZ2Rpr0faOMYAfAgjk2OSZaEc/edit?tab=t.0 for some analysis.
We currently map from write bytes to flushed bytes directly using a linear model `ax + b` (minimizing b, while holding `a` in the interval [0.5, 3.0]. The assumption here is that write bytes are to the raft log, and there will be a roughly 2x multiplier when also considering state machine application, and then there will be some reduction due to raft log truncation (before flush) and due to compression. The hard bounds on `a` exist to prevent a bad sample from skewing the model: even though we exponentially smooth `a` and `b`, the exponential smoothing multiplier is 0.5 so it can take some time for the model to get back to a good state.
This scheme started suffering from strain with WAL failover since all the write bytes do not get flushed in the 15s interval we use for the model estimation. We've introduced some conditionals around WAL failover being active to avoid updating the model, but this is deficient as since then we have also raised the memtable write stall limit to 16 memtables (when memory permits).
In the aforementioned issue, we are seeing probably > 50x compression when flushing, so the model over-estimates the tokens consumed since it reaches its lowest value of 0.5x+1 (it should actually be closer to 0.02x+1). Simply loosening the bounds on `a` is not the answer, due to the strain mentioned in the previous paragraph.
Instead we should be modeling this in two stages:
- write bytes => WAL bytes: this will account for the multiplier due to state machine application.
- WAL bytes flushed => L0 flushed bytes: Since the LHS is the actual WAL bytes flushed in that interval, there is no uncertainty on the size of the flush backlog that can cause model inaccuracy. This model will account for raft log truncation and compression. We can let the bounds of `a` in this model to be more free, say lower bound of 0.01.
We expect both these models to be more stable.
Jira issue: CRDB-56235
Contributor guide
Assessment
This issue has not been assessed yet.