aws / aws/amazon-cloudwatch-agent

Change Prometheus "instance" value

Open
#456 3 comments 1 reaction 0 assignees View on GitHub
aws/ecs component/prometheus enhancement
Dominant language
Go
Stars
550
Forks
271
Avg merge
1d 21h
Merged PRs (30d)
13

Description

**Is your feature request related to a problem? Please describe.**
For a Fargate task, the CW Agent for Prometheus emits `instance` to be the VPC IP address (plus port) of the container. For forensics investigation (ie. which task instance is this that is complaining?), this is not immediately helpful (through a number of queries, one could wander their way back to the task instance, but not as easily as knowing the task instance already.)

For example from a `cwagent-ecs-file-sd-config` log entry:
```
...
"instance": "172.31.14.84:9404"
...
```

**Describe the solution you'd like**
For the `instance` value, it would be nice for it to be the task instance id (e.g `7bd02f006a93486ab9bdc77e86c2bd68` - not the Task Definition); if there can exist a 1-N mapping of task instance and container instance, then the `instance` value would be better as the Container Runtime Id, as opposed to the VPC IP address.

**Describe alternatives you've considered**
Post processing of the IP address can be done, but for the Cloud Watch graph experience lending itself to immediate container drill-down, i am ignorant of a nice pain-free way to do this.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the ECS file service discovery path represented by the `cwagent-ecs-file-sd-config` log entry and how Prometheus assigns the `instance` value. Check whether the task instance ID or container runtime ID is available there; done means the emitted label supports immediate task-level identification instead of the VPC IP and port.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, prometheus
Domain
cloud, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.