cockroachdb / cockroachdb/cockroach
LDR: Elastic CPU Utilization is Undercounted For LDR
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
LDR opts into elastic admission control by using the bulk low priority request priority. The measured "elastic cpu utilization" is dramatically undercounted. In a 25.2 test cluster that was only serving LDR replication traffic, the real CPU usage was ~90% and the measured CPU "elastic" usage was only 10-12%.
Several components consume CPU in response to an LDR replicated event:
1. Decoding the event in the LDR processor and marshaling a KV request to write it.
2. Generic RPC handling, including gRPC and protobuf unmarshaling.
3. Handling the KV CPUT on the leaseholder.
4. Replicating the KV event to other nodes in the Raft group.
5. Flushing the memtable containing the event and compacting the write.
Of these, only # 3 (the KV request handler) is counted as elastic CPU usage. While this undercounting isn't ideal, it is generally acceptable because the actual value of a "token" in the elastic CPU token bucket is not critical. If elastic usage is undercounted, the system experiences scheduling latency at a lower utilization threshold and correspondingly reduces token allocation to elastic operations. However, increasing the lower bound on elastic CPU tokens causes the system to limit usage based on the measured (undercounted) elastic usage, rather than at the point where scheduling latency actually occurs.
One potential small improvement area for LDR is #1: integrating the processors with elastic admission control. However, because most of the work lies in # 4 and # 5, elastic CPU usage would still be undercounted.
To properly fix this we need:
1. A better model for elastic admission control cpu time consumed by small operations.
2. Somehow integrate elastic admission accounting with raft replication. Elastic writes replicated via raft should be accounted as elastic cpu usage on the receiving node.
Jira issue: CRDB-49879
Contributor guide
Assessment
This issue has not been assessed yet.