influxdata / influxdata/telegraf
Prometheus plugin - Kafka metrics is missing meta labels as tags
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Relevant telegraf.conf
```toml
[[inputs.prometheus]]
monitor_kubernetes_pods = false
response_timeout = "5s"
metric_version = 2
kubernetes_services = [
"http://kafka-metrics.default:9308/metrics",
"http://kafka-jmx-metrics.default:5556/metrics"
]
bearer_token = "/run/secrets/kubernetes.io/serviceaccount/token"
insecure_skip_verify = true
```
### Logs from Telegraf
```text
2022-07-19T14:37:11Z I! Starting Telegraf 1.23.2
2022-07-19T14:37:11Z I! Loaded inputs: internal prometheus
2022-07-19T14:37:11Z I! Loaded aggregators:
2022-07-19T14:37:11Z I! Loaded processors:
2022-07-19T14:37:11Z I! Loaded outputs: influxdb_v2 (2x)
2022-07-19T14:37:11Z I! Tags enabled: host=telegraf-prometheus-kafka-77d6b657c-zvmxl
2022-07-19T14:37:11Z I! [agent] Config: Interval:30s, Quiet:false, Hostname:"telegraf-prometheus-kafka-77d6b657c-zvmxl", Flush Interval:10s
2022-07-19T14:37:11Z D! [agent] Initializing plugins
2022-07-19T14:37:11Z D! [agent] Connecting outputs
2022-07-19T14:37:11Z D! [agent] Attempting connection to [outputs.influxdb_v2]
2022-07-19T14:37:11Z D! [agent] Successfully connected to outputs.influxdb_v2
2022-07-19T14:37:11Z D! [agent] Attempting connection to [outputs.influxdb_v2]
2022-07-19T14:37:11Z D! [agent] Successfully connected to outputs.influxdb_v2
2022-07-19T14:37:11Z D! [agent] Starting service inputs
2022-07-19T14:37:11Z I! Config watcher started
2022-07-19T14:37:21Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 50000 metrics
2022-07-19T14:37:21Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 50000 metrics
2022-07-19T14:37:31Z D! [outputs.influxdb_v2] Wrote batch of 50 metrics in 122.566406ms
2022-07-19T14:37:31Z D! [outputs.influxdb_v2] Buffer fullness: 199 / 50000 metrics
2022-07-19T14:37:31Z D! [outputs.influxdb_v2] Wrote batch of 50 metrics in 453.601643ms
2022-07-19T14:37:31Z D! [outputs.influxdb_v2] Buffer fullness: 199 / 50000 metrics
2022-07-19T14:37:41Z D! [outputs.influxdb_v2] Wrote batch of 199 metrics in 29.847702ms
2022-07-19T14:37:41Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 50000 metrics
2022-07-19T14:37:41Z D! [outputs.influxdb_v2] Wrote batch of 199 metrics in 231.098462ms
2022-07-19T14:37:41Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 50000 metrics
2022-07-19T14:37:51Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 50000 metrics
2022-07-19T14:37:51Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 50000 metrics
2022-07-19T14:38:01Z D! [outputs.influxdb_v2] Wrote batch of 247 metrics in 13.764863ms
2022-07-19T14:38:01Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 50000 metrics
2022-07-19T14:38:01Z D! [outputs.influxdb_v2] Wrote batch of 247 metrics in 233.323289ms
2022-07-19T14:38:01Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 50000 metrics
2022-07-19T14:38:11Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 50000 metrics
```
### System info
Telegraf 1.23.2, AWS EKS 1.20, Kafka installed via Bitnami Helm chart
### Docker
_No response_
### Steps to reproduce
Metric example I got from Prometheus server via ServiceMonitor:
```
kafka_controller_controllerchannelmanager_queuesize_value{broker_id="1", container="jmx-exporter", endpoint="http-metrics", instance="172.30.2.222:5556", job="kafka-jmx-metrics", namespace="default", pod="kafka-1", service="kafka-jmx-metrics"}
```
Prometheus metric example I got from the kafka metric exporter service endpoint:
```
\# HELP kafka_controller_controllerchannelmanager_totalqueuesize_value Attribute exposed for management kafka.controller:name=TotalQueueSize,type=ControllerChannelManager,attribute=Value
\# TYPE kafka_controller_controllerchannelmanager_totalqueuesize_value untyped
kafka_controller_controllerchannelmanager_totalqueuesize_value 0.0
```
Influxdb lineprotocol example from telegraf I got:
```
prometheus,address=10.100.4.63,host=telegraf-prometheus-kafka-77d6b657c-zvmxl,url=http://kafka-jmx-metrics.default:5556/metrics kafka_controller_controllerchannelmanager_totalqueuesize_value=0 1556813561098000000
```
### Expected behavior
Additional tags like broker_id="1", container="jmx-exporter", endpoint="http-metrics", instance="172.30.2.222:5556", job="kafka-jmx-metrics", namespace="default", pod="kafka-1", service="kafka-jmx-metrics".
### Actual behavior
I'm missing additional meta tags like broker_id="1", container="jmx-exporter", endpoint="http-metrics", instance="172.30.2.222:5556", job="kafka-jmx-metrics", namespace="default", pod="kafka-1", service="kafka-jmx-metrics". With current output I'm not able to distinguish between different Kafka brokers and all broker metrics are mixed together.
Is it possible to add prometheus meta labels as tags in prometheus plugin? Maybe it can be done with some configuration which I've missed.
### Additional info
[kafka-metrics-exporter-list.txt](https://github.com/influxdata/telegraf/files/9160401/kafka-metrics-exporter-list.txt)
Contributor guide
Research direction
Start at the Prometheus input plugin and compare the ServiceMonitor metric with the exporter metric shown in the issue. Verify how the plugin handles Prometheus labels, then ensure the expected labels are preserved as tags so metrics from different Kafka brokers can be distinguished.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, prometheus
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100