influxdata / influxdata/telegraf
Support additional, custom AWS resource tags for CloudWatch metrics
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
## Feature Request
We would like to enrich metrics from the AWS CloudWatch input about EC2 resources with their respective tags. However, the current plugin `aws_ec2` does not support this. From what I understand it is using the [AWS Instance Metadata](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) to detect which instance it's running on and selects the tags based on that. In our case Telegraf runs on a dedicated VM and collects the metrics from CloudWatch, therefore, the existing plugin cannot be used to collect tags.
### Proposal:
Have a way to enrich metrics that are coming from CloudWatch with tags from the resource the metric is from. Either as part of the existing CloudWatch plugin or as an additional processor (which would heavily rely on the CloudWatch input plugin).
### Current behavior:
Metrics from AWS CloudWatch only contain the resource name, instance id and region:
`cloudwatch_aws_nat_gateway,nat_gateway_id=nat-0a99753733657e0f5,region=eu-central-1 active_connection_count_sum=198 1644743280000000000`
### Desired behavior:
Metrics from AWS CloudWatch contain arbitrary, configurable labels from the origin resource (as well as resource name, instance id and region) e.g. `foo=bar`:
`cloudwatch_aws_nat_gateway,foo=bar,nat_gateway_id=nat-0a99753733657e0f5,region=eu-central-1 active_connection_count_sum=198 1644743280000000000`
### Use case:
Many instances of the same resource run in a single account and region. Being able to divide the metrics from them into groups by custom tags would greatly improve the observability of the individual deployments a resource belongs to.
### Steps taken:
As a first step I copied the existing `aws_ec2` plugin and stripped all the instance identity stuff from it. It now collects the instance id from the given metric and adds any configured tags. [See POC in my fork](https://github.com/maxmoehl/telegraf/commit/20f28b9d011a2aec50065c79ef8607a938ed49f1)
Edit: I implemented a tag cache that does most of the required work, but still has some open questions.
I'm happy to work on this and provide a solution (this time [with a signed CLA](https://github.com/influxdata/telegraf/pull/10123) :D). I'm not sure whether a new plugin or extending the existing CloudWatch plugin would be more desirable, I'm open for suggestions.
Contributor guide
Research direction
Start by comparing the existing aws_ec2 plugin with the CloudWatch input and review the linked proof of concept. Decide whether configurable resource-tag enrichment belongs in the CloudWatch plugin or a processor, then define the tag-cache behavior and verify that metrics include the configured tags alongside the existing resource fields.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100