Monitor a cluster's capacity in real-time

[This may be an open problem and has no immediate action to solve it.]

Knowing a cluster's capacity (e.g., throughput) and the utilization of the capacity in real-time can help applications to decide when the cluster is going to be saturated. This helps applications to better control (or throttle) their workload on the cluster.

However, a cluster's capacity is hard to calculate or monitor.
It is affected by several factors: (1) users' access pattern (read-only, write-only, read-and-write, or rangeread, and the key space accessed; (2) hardware status (e.g., when SSD firmware does garbage collection after a lot of reads/writes, the storage server's performance will be affected); (3) cluster maintenance (e.g., changing disk or rack-related maintenance).

The current solution in FDB is to estimate a cluster's capacity in the worst-case scenario. Applications uses the worst-case capacity estimation to throttle its workload.

An immediate action may be differentiating the capacity in different worst-case scenarios: such as no-maintenance vs. maintenance.

The issue names no files, tests, or entry points. Start by locating FoundationDB's current worst-case capacity estimation and understanding how maintenance and workload patterns affect it; completion would require a concrete, actionable design for real-time capacity monitoring or differentiated capacity estimates.

Written by the indexing model from the issue text.