telemetry: emit metrics about the size of raft commits
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
Currently we only emit metrics about the timing of raft commits (i.e. how long they take to commit and the rate) as well as the number of raft logs batched into each group that's appended as one.
In several incident responses recently we've had hypotheses about clients (mainly Vault) performing larger transactions and so causing more IO on server disks even though the overall rate is lower, however these is no way to confirm this currently.
We should add metrics (to the raft lib but captured here as a top-level issue for Consul specifically):
- **Summary of the _size_ of raft log entries being appended.** This would help analyse if the amount of data being written is larger even if the transaction rate is the same.
- **Summary of the _number of operations_ inside each `Txn`.** This would help understand how much actual work Vault or other clients are doing when they are batching it via Transactions.
Contributor guide
Research direction
Start in the Consul raft integration and trace the existing raft commit timing, rate, and batching metrics. Then inspect how raft log entries and Txn operations pass through the raft library; done means metrics summarize appended entry sizes and the number of operations in each transaction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100