opensearch-project / opensearch-project/data-prepper

OTel Trace from kafka source with otel_trace_raw

Open
#5,446 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
374
Forks
355
Avg merge
3d 18h
Merged PRs (30d)
8

Description

I have set up tracing in the method Otelcol -> DataPrepper -> OpenSearch, where Kafka is used as a buffer in DataPrepper. This method works completely and there are no issues with it.
After that, I needed to remove the Kafka buffer and use Kafka as a trace source instead, where they are written using otelcol.
Configuration:

entry-pipeline:
  source:
    kafka:
      bootstrap_servers:
        - kafka_host:9092
      topics:
        - name: test_trace_topic
          group_id: trace-test
      encryption:
        type: none 
  sink:
    - pipeline:
        name: "raw-trace-pipeline"
    - pipeline:
        name: "service-map-pipeline"

raw-trace-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  buffer:
    bounded_blocking:
       buffer_size: 65536
       batch_size: 8
  processor:
    - otel_trace_raw:
    - otel_trace_group:
        hosts: ["http://OpenSearch_host"]
        insecure: true
        username: admin
        password: password
  sink:
    - opensearch:
        hosts: ["http://OpenSearch_host"]
        insecure: true
        username: admin
        password: password
        index_type: trace-analytics-raw

service-map-pipeline:
  delay: "100"
  source:
    pipeline:
      name: "entry-pipeline"
  buffer:
    bounded_blocking:
      buffer_size: 65536
      batch_size: 8
  processor:
    - otel_trace_raw:
    - otel_trace_group:
        hosts: ["http://OpenSearch_host"]
        insecure: true
        username: admin
        password: password
  sink:
    - opensearch:
        hosts: ["http://OpenSearch_host"]
        insecure: true
        username: admin
        password: password
        index_type: trace-analytics-raw

service-map-pipeline:
  delay: "100"
  source:
    pipeline:
      name: "entry-pipeline"
  buffer:
    bounded_blocking:
      buffer_size: 65536
      batch_size: 8
  processor:
    - service_map_stateful:
  sink:
    - opensearch:
        hosts: ["http://OpenSearch_host"]
        insecure: true
        username: admin
        password: password
        index_type: trace-analytics-service-map

At the execution stage of the
processor - otel_trace_raw,
DataPrepper reports processing errors related to parsing.
log of the error:

2025-02-20T15:56:09,339 [raw-trace-pipeline-processor-worker-3-thread-1] ERROR org.opensearch.dataprepper.pipeline.ProcessWorker - A processor threw an exception. This batch of Events will be dropped, and their EventHandles will be released: 
java.lang.ClassCastException: class org.opensearch.dataprepper.model.log.JacksonLog cannot be cast to class org.opensearch.dataprepper.model.trace.Span (org.opensearch.dataprepper.model.log.JacksonLog and org.opensearch.dataprepper.model.trace.Span are in unnamed module of loader 'app')
        at org.opensearch.dataprepper.plugins.processor.oteltrace.OTelTraceRawProcessor.doExecute(OTelTraceRawProcessor.java:89) ~[otel-trace-raw-processor-2.10.1.jar:?]
        at org.opensearch.dataprepper.model.processor.AbstractProcessor.lambda$execute$0(AbstractProcessor.java:54) ~[data-prepper-api-2.10.1.jar:?]
        at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:69) ~[micrometer-core-1.13.0.jar:1.13.0]
        at org.opensearch.dataprepper.model.processor.AbstractProcessor.execute(AbstractProcessor.java:54) ~[data-prepper-api-2.10.1.jar:?]
        at org.opensearch.dataprepper.peerforwarder.PeerForwardingProcessorDecorator.execute(PeerForwardingProcessorDecorator.java:103) ~[data-prepper-core-2.10.1.jar:?]
        at org.opensearch.dataprepper.pipeline.ProcessWorker.doRun(ProcessWorker.java:139) [data-prepper-core-2.10.1.jar:?]
        at org.opensearch.dataprepper.pipeline.ProcessWorker.run(ProcessWorker.java:61) [data-prepper-core-2.10.1.jar:?]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
2025-02-20T15:56:09,426 [service-map-pipeline-processor-worker-5-thread-1] ERROR org.opensearch.dataprepper.pipeline.ProcessWorker - A processor threw an exception. This batch of Events will be dropped, and their EventHandles will be released: 
java.lang.ClassCastException: class org.opensearch.dataprepper.model.event.JacksonEvent cannot be cast to class org.opensearch.dataprepper.model.trace.Span (org.opensearch.dataprepper.model.event.JacksonEvent and org.opensearch.dataprepper.model.trace.Span are in unnamed module of loader 'app')
        at org.opensearch.dataprepper.plugins.processor.ServiceMapStatefulProcessor.lambda$doExecute$5(ServiceMapStatefulProcessor.java:152) ~[service-map-stateful-2.10.1.jar:?]
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
        at org.opensearch.dataprepper.plugins.processor.ServiceMapStatefulProcessor.doExecute(ServiceMapStatefulProcessor.java:152) ~[service-map-stateful-2.10.1.jar:?]
        at org.opensearch.dataprepper.model.processor.AbstractProcessor.lambda$execute$0(AbstractProcessor.java:54) ~[data-prepper-api-2.10.1.jar:?]
        at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:69) ~[micrometer-core-1.13.0.jar:1.13.0]
        at org.opensearch.dataprepper.model.processor.AbstractProcessor.execute(AbstractProcessor.java:54) ~[data-prepper-api-2.10.1.jar:?]
        at org.opensearch.dataprepper.peerforwarder.PeerForwardingProcessorDecorator.execute(PeerForwardingProcessorDecorator.java:103) ~[data-prepper-core-2.10.1.jar:?]
        at org.opensearch.dataprepper.pipeline.ProcessWorker.doRun(ProcessWorker.java:139) [data-prepper-core-2.10.1.jar:?]
        at org.opensearch.dataprepper.pipeline.ProcessWorker.run(ProcessWorker.java:61) [data-prepper-core-2.10.1.jar:?]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

Is there a solution to this problem, because DataPrepper can work with data obtained from Kafka when it is used at the buffer stage, but it cannot work with the same data when it is obtained at the source stage? As a result, the situation looks like a bug.

Environment (please complete the following information):

  • OS: [Debian 12]
  • Version DataPrepper [2.10.1]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with OTelTraceRawProcessor.java:89 and ServiceMapStatefulProcessor.java:152, then compare how the Kafka source and buffer paths represent events in Data Prepper 2.10.1. Reproduce the provided configuration and ClassCastException, and inspect the related processor tests if present. Done means Kafka-sourced traces can pass through both pipelines without the reported cast errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.