opensearch-project / opensearch-project/data-prepper
Errors in the collector when configuring Prometheus as the receiver and Data Prepper as the exporter.
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
I configured the opentelemetry collector with Prometheus as the receiver and data prepper as the exporter. However, I'm encountering no error in the collector, and no data is showing up in the OpenSearch dashboard. What should I do? What might I have done wrong?
I'll attach the Data Prepper configmap YAML and the OpenTelemetry Collector configmap YAML below.
[data prepper configmap yaml]
otel-metrics-pipeline-2:
# workers: 5
delay: 10
source:
http_source:
ssl: false
port: 21891
buffer:
bounded_blocking:
buffer_size: 12800
batch_size: 1024
sink:
- opensearch:
hosts: ["https://opensearch-cluster-master.opensearch.svc:9200"]
insecure: true
username: admin
password: admin
index_type: custom
index: proms-%{yyyy.MM.dd}
[otel collector configmap yaml]
data:
relay: |
exporters:
debug: {}
logging: {}
otlp/metrics:
endpoint: data-prepper-headless.opensearch.svc.cluster.local:21891
tls:
insecure: true
extensions:
health_check:
endpoint: ${env:K8S_POD_IP}:13133
processors:
memory_limiter:
check_interval: 5s
limit_percentage: 80
spike_limit_percentage: 25
receivers:
prometheus/internal:
config:
scrape_configs:
- job_name: apps
kubernetes_sd_configs:
- role: pod
selectors:
- role: pod
# only scrape data from pods running on the same node as collector
field: "spec.nodeName=${NODE_NAME}"
relabel_configs:
# scrape pods annotated with "prometheus.io/scrape: true"
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
regex: "true"
action: keep
# read the port from "prometheus.io/port: <port>" annotation and update scraping address accordingly
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: ([^:]+)(?::\d+)?;(\d+)
# escaped $1:$2
replacement: $$1:$$2
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
- job_name: 'otel-collector'
scrape_interval: 5s
static_configs:
- targets: ['prometheus-k8s.monitoring.svc:9090']
metric_relabel_configs:
- source_labels: [ __name__ ]
regex: '.*grpc_io.*'
action: drop
otlp:
protocols:
grpc:
endpoint: ${env:K8S_POD_IP}:4317
http:
endpoint: ${env:K8S_POD_IP}:4318
service:
extensions:
- health_check
pipelines:
metrics/internal:
exporters:
- debug
- logging
- otlp/metrics
processors:
- memory_limiter
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the attached Data Prepper and OpenTelemetry Collector ConfigMap YAML, starting with the receiver/exporter endpoint and metrics pipeline wiring. Reproduce the Prometheus-to-Data Prepper path and inspect collector, Data Prepper, and OpenSearch output. Done means identifying the configuration issue and documenting the settings needed for metrics to appear.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, prometheus
- Domain
- data-engineering, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100