hashicorp / hashicorp/consul

telemetry: emit metrics about the size of raft commits

Open
#10,215 0 comments 0 reactions 0 assignees View on GitHub
theme/reliability theme/telemetry type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

Currently we only emit metrics about the timing of raft commits (i.e. how long they take to commit and the rate) as well as the number of raft logs batched into each group that's appended as one.

In several incident responses recently we've had hypotheses about clients (mainly Vault) performing larger transactions and so causing more IO on server disks even though the overall rate is lower, however these is no way to confirm this currently.

We should add metrics (to the raft lib but captured here as a top-level issue for Consul specifically):

- **Summary of the _size_ of raft log entries being appended.** This would help analyse if the amount of data being written is larger even if the transaction rate is the same.
- **Summary of the _number of operations_ inside each `Txn`.** This would help understand how much actual work Vault or other clients are doing when they are batching it via Transactions.

Contributor guide

Open the contributing guide

Research direction

Start in the Consul raft integration and trace the existing raft commit timing, rate, and batching metrics. Then inspect how raft log entries and Txn operations pass through the raft library; done means metrics summarize appended entry sizes and the number of operations in each transaction.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.