opensearch-project / opensearch-project/data-prepper

Improved otel_metrics_source documentation

Open
#1,791 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.
In Data Prepper v1.5.1 the otel_metrics_source cannot be used with the string_converter plugin.

Describe the solution you'd like
Documentation on plugin/feature compatibility.

Describe alternatives you've considered (Optional)
Enhance pipeline validation on startup to detect incomatibility issues.

Additional context

Related community forum question

pipelines.yaml
entry-pipeline:
  source:
    otel_metrics_source:
      ssl: false
  sink:
    - stdout:
otel-collector-config.yaml
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'data-prepper'
          scrape_interval: 10s
          metrics_path: "/metrics/prometheus"
          static_configs:
            - targets: ['host.docker.internal:4900']

exporters:
  otlp/2:
    endpoint: data-prepper:21891
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [otlp/2]
prometheus.yaml
docker-compose.yaml
version: "3.9"

services:
  data-prepper:
    restart: unless-stopped
    image: opensearch-data-prepper:2.0.0-SNAPSHOT
    working_dir: /usr/share/data-prepper/
    volumes:
      - ./data-prepper/config/data-prepper-config.yaml:/usr/share/data-prepper/config/data-prepper-config.yaml
      - ./data-prepper/config/pipelines.yaml:/usr/share/data-prepper/pipelines.yaml
      - ../../shared-config/log4j2.properties:/usr/share/data-prepper/config/log4j.properties
    ports:
      - "2021:2021"
  otel-collector:
    restart: unless-stopped
    image: otel/opentelemetry-collector:0.40.0
    command: ["--config=/etc/otel-collector-config.yml"]
    volumes:
      - ./opentelemetry-collector/otel-collector-config.yml:/etc/otel-collector-config.yml
    depends_on:
      - data-prepper
    ports:
      - "4317:4317"
  prometheus:
    restart: unless-stopped
    image: prom/prometheus
    volumes:
      - ./prometheus/prometheus.yaml:/etc/prometheus/prometheus.yaml
    ports:
      - "9000:9000"
    depends_on:
      - otel-collector

I recommend using the smoke-tests docker-compose.yaml as a base for setup.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the smoke-tests docker-compose.yaml and the example pipelines.yaml, including the otel_metrics_source and string_converter configuration described here. Check the existing plugin documentation for compatibility guidance. Done means documenting the incompatibility clearly, with the relevant configuration context and any supported alternatives.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose
Domain
documentation, observability
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.