aws-samples / aws-samples/amazon-cloudwatch-container-insights

CloudWatch Prometheus agent can't read metrics export

Open
#34 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
178
Forks
117
Avg merge
1d 10h
Merged PRs (30d)
3

Description

The Cloudwatch prometheus agent is unable to read or determine the type of exported metric data:
```020-08-03T22:00:05Z D! [96/500] Unsupported Prometheus metric: kamailio_sl_1xx_replies with type:
2020-08-03T22:00:05Z D! [97/500] Unsupported Prometheus metric: kamailio_sl_200_replies with type:
2020-08-03T22:00:05Z D! [98/500] Unsupported Prometheus metric: kamailio_sl_202_replies with type:
2020-08-03T22:00:05Z D! [99/500] Unsupported Prometheus metric: kamailio_sl_2xx_replies with type:
2020-08-03T22:00:05Z D! [100/500] Unsupported Prometheus metric: kamailio_sl_300_replies with type:
2020-08-03T22:00:05Z D! [101/500] Unsupported Prometheus metric: kamailio_sl_301_replies with type:
2020-08-03T22:00:05Z D! [102/500] Unsupported Prometheus metric: kamailio_sl_302_replies with type:
2020-08-03T22:00:05Z D! [103/500] Unsupported Prometheus metric: kamailio_sl_3xx_replies with type:
2020-08-03T22:00:05Z D! [104/500] Unsupported Prometheus metric: kamailio_sl_400_replies with type:
2020-08-03T22:00:05Z D! [105/500] Unsupported Prometheus metric: kamailio_sl_401_replies with type:
2020-08-03T22:00:05Z D! [106/500] Unsupported Prometheus metric: kamailio_sl_403_replies with type:
2020-08-03T22:00:05Z D! [107/500] Unsupported Prometheus metric: kamailio_sl_404_replies with type:
2020-08-03T22:00:05Z D! [108/500] Unsupported Prometheus metric: kamailio_sl_407_replies with type:
2020-08-03T22:00:05Z D! [109/500] Unsupported Prometheus metric: kamailio_sl_408_replies with type:
2020-08-03T22:00:05Z D! [110/500] Unsupported Prometheus metric: kamailio_sl_483_replies with type:
2020-08-03T22:00:05Z D! [111/500] Unsupported Prometheus metric: kamailio_sl_4xx_replies with type:
2020-08-03T22:00:05Z D! [112/500] Unsupported Prometheus metric: kamailio_sl_500_replies with type:
2020-08-03T22:00:05Z D! [113/500] Unsupported Prometheus metric: kamailio_sl_5xx_replies with type:
```
The metric data is exported in counter format with no labels:
```kamailio_sl_1xx_replies 0 1596492670513
kamailio_sl_200_replies 12 1596492670513
kamailio_sl_202_replies 0 1596492670513
kamailio_sl_2xx_replies 0 1596492670513
kamailio_sl_300_replies 0 1596492670513
kamailio_sl_301_replies 0 1596492670513
kamailio_sl_302_replies 0 1596492670513
kamailio_sl_3xx_replies 0 1596492670513
kamailio_sl_400_replies 0 1596492670513
kamailio_sl_401_replies 0 1596492670513
kamailio_sl_403_replies 0 1596492670513
kamailio_sl_404_replies 0 1596492670513
kamailio_sl_407_replies 0 1596492670513
kamailio_sl_408_replies 0 1596492670513
kamailio_sl_483_replies 0 1596492670513
kamailio_sl_4xx_replies 0 1596492670513
```

The export data can be read by a stand alone Prometheus, I used the latest docker image prometheus:v2.19.3.

Is this export format not valid for the current version of the Cloud Watch Prometheus agent?

Contributor guide

Open the contributing guide

Research direction

Start with the reported metric export and the CloudWatch Prometheus agent's handling of counter metrics without labels, then compare the result with the standalone Prometheus v2.19.3 behavior. Done means determining whether this export is valid for the agent and, if it is, confirming that these metrics are read without the unsupported-type errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, prometheus
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.