opensearch-project / opensearch-project/data-prepper

When I configured the Prometheus receiver, the fields were not differentiated.

Open
#4,302 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

I have configured the otel collector with Prometheus as the receiver and Data Prepper as the exporter.
So, while I can retrieve data from the OpenSearch dashboard, it appears that all data is grouped under the name field.
To create a dashboard, I believe the fields need to be distinguished.
What additional settings should I add to the otel collector configmap to differentiate fields?
I am attaching the yaml file of the otel collector configmap that I have written. Thank you.

[otel collector configmap yaml]
data:
relay: |
exporters:
otlp/metrics:
endpoint: data-prepper-headless.opensearch.svc:21891
tls:
insecure: true
extensions:
health_check:
endpoint: ${env:K8S_POD_IP}:13133
processors:
memory_limiter:
check_interval: 5s
limit_percentage: 80
spike_limit_percentage: 25
receivers:
prometheus/internal:
config:
scrape_configs:
- job_name: apps
kubernetes_sd_configs:
- role: pod
selectors:
- role: pod
# only scrape data from pods running on the same node as collector
field: "spec.nodeName=${NODE_NAME}"
relabel_configs:
# scrape pods annotated with "prometheus.io/scrape: true"
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
regex: "true"
action: keep
# read the port from "prometheus.io/port: " annotation and update scraping address accordingly
- source_labels: [address, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
target_label: address
regex: ([^:]+)(?::\d+)?;(\d+)
# escaped $1:$2
replacement: $$1:$$2
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
- job_name: 'otel-collector'
scrape_interval: 5s
static_configs:
- targets: ['prometheus-k8s.monitoring.svc:9090']
metric_relabel_configs:
- source_labels: [ name ]
regex: '.grpc_io.'
action: drop
otlp:
protocols:
grpc:
endpoint: ${env:K8S_POD_IP}:4317
http:
endpoint: ${env:K8S_POD_IP}:4318
service:
extensions:
- health_check
pipelines:
metrics/internal:
exporters:
- otlp/metrics
processors:
- memory_limiter
receivers:
- prometheus/internal
- otlp

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the attached otel collector configmap, especially the prometheus/internal receiver and metrics/internal pipeline that sends data through the OTLP exporter to Data Prepper. Compare the resulting fields with the OpenSearch dashboard output and determine which configuration changes would distinguish them; done means the exported metrics expose separate, usable fields for dashboard creation.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, prometheus
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.