add a few metrics for service health updates and streaming
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Feature Description
Add metrics to expose service health update rates, and help estimate streaming CPU usage.
We did some performance tests on consul service health streaming and found that streaming CPU usage is proportional to the total number of update events a server sends to all streaming clients, that is:
$$ CPU = k \times \sum_{services}serviceStreams \times servcieUpdateRate $$
The following three metrics could be beneficial for the above:
1. A counter: `consul.health.service.update_count` with service name as a label. This counter increments for each service health update.
2. A gauge: `consul.health.service.streams` with service name as a label. Tracks the number of stream subscriptions to a service.
3. A counter: `consul.health.stream_events.total_sent`. This counter increments for each invocation of `serverStream.Send(e)`.
#### Use Case(s)
The metric `consul.health.stream_events.total_sent` can give estimates of CPU usage of service health streaming. The metrics `consul.health.service.streams` and `consul.health.service.update_count` provides break down of the usage.
Additionally `consul.health.service.update_count` shows how stable the services are, and `consul.health.service.streams` shows the number of streaming clients for each service, which helps manage streaming clients.
Contributor guide
Research direction
Start by locating the Consul service health streaming implementation and the `serverStream.Send(e)` call mentioned in the issue. Trace where service health updates and stream subscriptions are handled, then verify the three named metrics for update counts, subscription counts, and total sent events, including their service labels where specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100