hashicorp / hashicorp/consul

add a few metrics for service health updates and streaming

Open
#14,324 1 comment 5 reactions 0 assignees View on GitHub
theme/telemetry type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Feature Description

Add metrics to expose service health update rates, and help estimate streaming CPU usage.

We did some performance tests on consul service health streaming and found that streaming CPU usage is proportional to the total number of update events a server sends to all streaming clients, that is:

$$ CPU = k \times \sum_{services}serviceStreams \times servcieUpdateRate $$

The following three metrics could be beneficial for the above:

1. A counter: `consul.health.service.update_count` with service name as a label. This counter increments for each service health update.
2. A gauge: `consul.health.service.streams` with service name as a label. Tracks the number of stream subscriptions to a service.
3. A counter: `consul.health.stream_events.total_sent`. This counter increments for each invocation of `serverStream.Send(e)`.

#### Use Case(s)

The metric `consul.health.stream_events.total_sent` can give estimates of CPU usage of service health streaming. The metrics `consul.health.service.streams` and `consul.health.service.update_count` provides break down of the usage.

Additionally `consul.health.service.update_count` shows how stable the services are, and `consul.health.service.streams` shows the number of streaming clients for each service, which helps manage streaming clients.

Contributor guide

Open the contributing guide

Research direction

Start by locating the Consul service health streaming implementation and the `serverStream.Send(e)` call mentioned in the issue. Trace where service health updates and stream subscriptions are handled, then verify the three named metrics for update counts, subscription counts, and total sent events, including their service labels where specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.