linkedin / linkedin/Burrow

Allow notifiers to receive metrics regardless of consumer group status

Open
#341 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

We currently have a working fork where we implemented two additional notifiers: one to push metrics to AWS CloudWatch [1] and another one to push metrics to dogstatsd [2](https://docs.datadoghq.com/developers/dogstatsd/).

Long story short, here at Vend we have a containerised driven architecture leveraged on AWS ECS and we wanted a way to be able to autoscale our consumers based on consumer lag. AWS Application autoscaling does not work well with sparse metric, which means that we need to push every single metric to cloudwatch regardless of the consumer status. As of [2], we push all metrics to dogstatsd so we can plot nice consumer lag graphs and also to create some monitors that may eventually trigger pagerduty if a consumer lag goes over a defined threshold.

Having said that ^^, at the moment there is no elegant way to enable `ShowAll` per notifier. Since we don't care about any other notifier we simply added `ShowAll: True` to https://github.com/linkedin/Burrow/blob/master/core/internal/notifier/coordinator.go#L375

```
go func(sendCluster string, sendConsumer string) {
nc.App.EvaluatorChannel <- &protocol.EvaluatorRequest{
Reply: nc.evaluatorResponse,
Cluster: sendCluster,
Group: sendConsumer,
ShowAll: true,
}
}(cluster, consumer)
```

The aforementioned approach is not ideal should we wish to push those two custom notifiers to upstream and on top of that it makes merging off upstream a bit cumbersome sometimes.

Please advise on what the best approach would be to address the above scenario.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in core/internal/notifier/coordinator.go around line 375 and trace how evaluator requests are created for notifiers. Determine how a notifier-specific ShowAll setting could support the CloudWatch and dogstatsd use cases without changing other notifiers. Done means each notifier can independently receive metrics regardless of consumer group status, with existing behavior preserved for others.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
backend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.