Dynamic write cache threading

Is your feature request related to a problem? Please describe.

I'm always frustrated when we can have better user experience but we don't yet. Our write cache is designed to be greedy in terms of used disk I/O and flushes things as fast as it can. But in a mixed RW/RO scenario this makes RO requests suffer a bit more than needed technically for short write load spikes. If this write burst completely fits into write cache size node can serve a bit more read requests while slowly flushing WC to the backing store.

Describe the solution you'd like

We can throttle WC flushing by limiting the number of threads involved. Some threads can be parked (waiting for signal) and only activated once WC hits some percentage of used space (like a single thread until 10% and then add threads linearly so that we'd have all of them doing their thing at 50% used space).

Notice that this will degrade pure writing load performance since WC will be depleted earlier in this case, but it's acceptable since scenarios with short write bursts and ~constant RO load are more probable in real life. This can be mitigated by increasing the size of WC as well.

No files, tests, or entry points are named. Start by locating the write-cache flushing and worker-thread implementation, then determine how thread activation could follow cache occupancy while preserving the proposed mixed RW/RO behavior. Done should include validation for short write bursts under read load and the noted pure-write performance trade-off.