Allow notifiers to receive metrics regardless of consumer group status
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
We currently have a working fork where we implemented two additional notifiers: one to push metrics to AWS CloudWatch [1] and another one to push metrics to dogstatsd [2](https://docs.datadoghq.com/developers/dogstatsd/).
Long story short, here at Vend we have a containerised driven architecture leveraged on AWS ECS and we wanted a way to be able to autoscale our consumers based on consumer lag. AWS Application autoscaling does not work well with sparse metric, which means that we need to push every single metric to cloudwatch regardless of the consumer status. As of [2], we push all metrics to dogstatsd so we can plot nice consumer lag graphs and also to create some monitors that may eventually trigger pagerduty if a consumer lag goes over a defined threshold.
Having said that ^^, at the moment there is no elegant way to enable `ShowAll` per notifier. Since we don't care about any other notifier we simply added `ShowAll: True` to https://github.com/linkedin/Burrow/blob/master/core/internal/notifier/coordinator.go#L375
```
go func(sendCluster string, sendConsumer string) {
nc.App.EvaluatorChannel <- &protocol.EvaluatorRequest{
Reply: nc.evaluatorResponse,
Cluster: sendCluster,
Group: sendConsumer,
ShowAll: true,
}
}(cluster, consumer)
```
The aforementioned approach is not ideal should we wish to push those two custom notifiers to upstream and on top of that it makes merging off upstream a bit cumbersome sometimes.
Please advise on what the best approach would be to address the above scenario.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in core/internal/notifier/coordinator.go around line 375 and trace how evaluator requests are created for notifiers. Determine how a notifier-specific ShowAll setting could support the CloudWatch and dogstatsd use cases without changing other notifiers. Done means each notifier can independently receive metrics regardless of consumer group status, with existing behavior preserved for others.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100