influxdata / influxdata/telegraf

Prometheus plugin - Kafka metrics is missing meta labels as tags

Open
#11,537 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Relevant telegraf.conf

```toml
[[inputs.prometheus]]
monitor_kubernetes_pods = false
response_timeout = "5s"
metric_version = 2
kubernetes_services = [
"http://kafka-metrics.default:9308/metrics",
"http://kafka-jmx-metrics.default:5556/metrics"
]
bearer_token = "/run/secrets/kubernetes.io/serviceaccount/token"
insecure_skip_verify = true
```

### Logs from Telegraf

```text
2022-07-19T14:37:11Z I! Starting Telegraf 1.23.2
2022-07-19T14:37:11Z I! Loaded inputs: internal prometheus
2022-07-19T14:37:11Z I! Loaded aggregators:
2022-07-19T14:37:11Z I! Loaded processors:
2022-07-19T14:37:11Z I! Loaded outputs: influxdb_v2 (2x)
2022-07-19T14:37:11Z I! Tags enabled: host=telegraf-prometheus-kafka-77d6b657c-zvmxl
2022-07-19T14:37:11Z I! [agent] Config: Interval:30s, Quiet:false, Hostname:"telegraf-prometheus-kafka-77d6b657c-zvmxl", Flush Interval:10s
2022-07-19T14:37:11Z D! [agent] Initializing plugins
2022-07-19T14:37:11Z D! [agent] Connecting outputs
2022-07-19T14:37:11Z D! [agent] Attempting connection to [outputs.influxdb_v2]
2022-07-19T14:37:11Z D! [agent] Successfully connected to outputs.influxdb_v2
2022-07-19T14:37:11Z D! [agent] Attempting connection to [outputs.influxdb_v2]
2022-07-19T14:37:11Z D! [agent] Successfully connected to outputs.influxdb_v2
2022-07-19T14:37:11Z D! [agent] Starting service inputs
2022-07-19T14:37:11Z I! Config watcher started
2022-07-19T14:37:21Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 50000 metrics
2022-07-19T14:37:21Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 50000 metrics
2022-07-19T14:37:31Z D! [outputs.influxdb_v2] Wrote batch of 50 metrics in 122.566406ms
2022-07-19T14:37:31Z D! [outputs.influxdb_v2] Buffer fullness: 199 / 50000 metrics
2022-07-19T14:37:31Z D! [outputs.influxdb_v2] Wrote batch of 50 metrics in 453.601643ms
2022-07-19T14:37:31Z D! [outputs.influxdb_v2] Buffer fullness: 199 / 50000 metrics
2022-07-19T14:37:41Z D! [outputs.influxdb_v2] Wrote batch of 199 metrics in 29.847702ms
2022-07-19T14:37:41Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 50000 metrics
2022-07-19T14:37:41Z D! [outputs.influxdb_v2] Wrote batch of 199 metrics in 231.098462ms
2022-07-19T14:37:41Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 50000 metrics
2022-07-19T14:37:51Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 50000 metrics
2022-07-19T14:37:51Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 50000 metrics
2022-07-19T14:38:01Z D! [outputs.influxdb_v2] Wrote batch of 247 metrics in 13.764863ms
2022-07-19T14:38:01Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 50000 metrics
2022-07-19T14:38:01Z D! [outputs.influxdb_v2] Wrote batch of 247 metrics in 233.323289ms
2022-07-19T14:38:01Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 50000 metrics
2022-07-19T14:38:11Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 50000 metrics
```

### System info

Telegraf 1.23.2, AWS EKS 1.20, Kafka installed via Bitnami Helm chart

### Docker

_No response_

### Steps to reproduce

Metric example I got from Prometheus server via ServiceMonitor:
```
kafka_controller_controllerchannelmanager_queuesize_value{broker_id="1", container="jmx-exporter", endpoint="http-metrics", instance="172.30.2.222:5556", job="kafka-jmx-metrics", namespace="default", pod="kafka-1", service="kafka-jmx-metrics"}
```

Prometheus metric example I got from the kafka metric exporter service endpoint:
```
\# HELP kafka_controller_controllerchannelmanager_totalqueuesize_value Attribute exposed for management kafka.controller:name=TotalQueueSize,type=ControllerChannelManager,attribute=Value
\# TYPE kafka_controller_controllerchannelmanager_totalqueuesize_value untyped
kafka_controller_controllerchannelmanager_totalqueuesize_value 0.0
```

Influxdb lineprotocol example from telegraf I got:
```
prometheus,address=10.100.4.63,host=telegraf-prometheus-kafka-77d6b657c-zvmxl,url=http://kafka-jmx-metrics.default:5556/metrics kafka_controller_controllerchannelmanager_totalqueuesize_value=0 1556813561098000000
```

### Expected behavior

Additional tags like broker_id="1", container="jmx-exporter", endpoint="http-metrics", instance="172.30.2.222:5556", job="kafka-jmx-metrics", namespace="default", pod="kafka-1", service="kafka-jmx-metrics".

### Actual behavior

I'm missing additional meta tags like broker_id="1", container="jmx-exporter", endpoint="http-metrics", instance="172.30.2.222:5556", job="kafka-jmx-metrics", namespace="default", pod="kafka-1", service="kafka-jmx-metrics". With current output I'm not able to distinguish between different Kafka brokers and all broker metrics are mixed together.

Is it possible to add prometheus meta labels as tags in prometheus plugin? Maybe it can be done with some configuration which I've missed.

### Additional info

[kafka-metrics-exporter-list.txt](https://github.com/influxdata/telegraf/files/9160401/kafka-metrics-exporter-list.txt)

Contributor guide

Open the contributing guide

Research direction

Start at the Prometheus input plugin and compare the ServiceMonitor metric with the exporter metric shown in the issue. Verify how the plugin handles Prometheus labels, then ensure the expected labels are preserved as tags so metrics from different Kafka brokers can be distinguished.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, prometheus
Domain
observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.