opensearch-project / opensearch-project/data-prepper

Errors in the collector when configuring Prometheus as the receiver and Data Prepper as the exporter.

Open
#4,299 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

I configured the opentelemetry collector with Prometheus as the receiver and data prepper as the exporter. However, I'm encountering no error in the collector, and no data is showing up in the OpenSearch dashboard. What should I do? What might I have done wrong?
I'll attach the Data Prepper configmap YAML and the OpenTelemetry Collector configmap YAML below.

[data prepper configmap yaml]

    otel-metrics-pipeline-2:
      #      workers: 5
      delay: 10
      source:
        http_source:
          ssl: false
          port: 21891
      buffer:
        bounded_blocking:
          buffer_size: 12800
          batch_size: 1024
      sink:
        - opensearch:
            hosts: ["https://opensearch-cluster-master.opensearch.svc:9200"]
            insecure: true
            username: admin
            password: admin
            index_type: custom
            index: proms-%{yyyy.MM.dd}

[otel collector configmap yaml]

data:
  relay: |
    exporters:
      debug: {}
      logging: {}
      otlp/metrics:
        endpoint: data-prepper-headless.opensearch.svc.cluster.local:21891
        tls:
          insecure: true
    extensions:
      health_check:
        endpoint: ${env:K8S_POD_IP}:13133
    processors:
      memory_limiter:
        check_interval: 5s
        limit_percentage: 80
        spike_limit_percentage: 25
    receivers:
      prometheus/internal:
        config:
          scrape_configs:
          - job_name: apps
            kubernetes_sd_configs:
            - role: pod
              selectors:
              - role: pod
                # only scrape data from pods running on the same node as collector
                field: "spec.nodeName=${NODE_NAME}"
            relabel_configs:
            # scrape pods annotated with "prometheus.io/scrape: true"
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              regex: "true"
              action: keep
              # read the port from "prometheus.io/port: <port>" annotation and update scraping address accordingly
            - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
              action: replace
              target_label: __address__
              regex: ([^:]+)(?::\d+)?;(\d+)
              # escaped $1:$2
              replacement: $$1:$$2
            - source_labels: [__meta_kubernetes_namespace]
              action: replace
              target_label: kubernetes_namespace
            - source_labels: [__meta_kubernetes_pod_name]
              action: replace
              target_label: kubernetes_pod_name
          - job_name: 'otel-collector'
            scrape_interval: 5s
            static_configs:
            - targets: ['prometheus-k8s.monitoring.svc:9090']
            metric_relabel_configs:
              - source_labels: [ __name__ ]
                regex: '.*grpc_io.*'
                action: drop
      otlp:
        protocols:
          grpc:
            endpoint: ${env:K8S_POD_IP}:4317
          http:
            endpoint: ${env:K8S_POD_IP}:4318
    service:
      extensions:
      - health_check
      pipelines:
        metrics/internal:
          exporters:
          - debug
          - logging
          - otlp/metrics
          processors:
          - memory_limiter

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the attached Data Prepper and OpenTelemetry Collector ConfigMap YAML, starting with the receiver/exporter endpoint and metrics pipeline wiring. Reproduce the Prometheus-to-Data Prepper path and inspect collector, Data Prepper, and OpenSearch output. Done means identifying the configuration issue and documenting the settings needed for metrics to appear.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, prometheus
Domain
data-engineering, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.