opensearch-project / opensearch-project/data-prepper

[BUG] add_entries processor add_when is not working when checking for fields within log.attributes or resource.attributes

Open
#5,279 7 comments 1 reaction 1 assignee View on GitHub

@san81 is already working on this.

Since Jan 7, 2025.

bug
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
The add_entries processor add_when is not working when checking for fields inside log.attributes or resource.attributes. It is working work when I check for a top level field such as severityText

I just setup data prepper docker container as mentioned in here

docker run --name data-prepper -p 2021:2021 -v /${PWD}/pipelines.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml opensearchproject/data-prepper:latest

This is my data prepper config:


otel-opensearch-pipeline:
  workers: 1
  delay: "5000"
  source:
    otel_logs_source:
      ssl: false
      port: 2021
      #compression: gzip
  processor:
    - add_entries:
        entries:
        - key: "add_entry_test"
          value: "done"
          add_when: /severityText == "Info"
        - key: "add_entry_dot_test"
          value: done
          add_when: /log.attributes.foo == "bar"    

    - date:
        from_time_received: true
        destination: "@timestamp"

  sink:
    - stdout:

Also in the same machine I have setup opentelemetry-collector-contrib:0.116.1 to receive OTEL logs and forward to data prepper

docker run -p 4317:4317 -v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml otel/opentelemetry-collector-contrib:0.116.1
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:

exporters:
  otlp:
    endpoint: "myhost:2021"
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

I am using Telemetry generator for OpenTelemetry to generate OTEL logs

./telemetrygen logs --body "2024-12-12:00:00:00 INFO This is a test message" --otlp-attributes host.name=\"mydevhost\" --telemetry-attributes foo=\"bar\" --trace-id ae87dadd90e9935a4bc9660628efd569 --span-id 5828fa4960140870 --duration 1s --otlp-insecure

OUTPUT

{
    "traceId": "ae87dadd90e9935a4bc9660628efd569",
    "spanId": "5828fa4960140870",
    "severityText": "Info",
    "flags": 0,
    "time": "2024-12-27T10:01:38.195840064Z",
    "severityNumber": 9,
    "droppedAttributesCount": 0,
    "serviceName": null,
    "body": "2024-12-12:00:00:00 INFO This is a test message",
    "observedTime": "1970-01-01T00:00:00Z",
    "schemaUrl": "https://opentelemetry.io/schemas/1.4.0",
    "add_entry_test": "done",
    "@timestamp": "2024-12-27T10:01:38.330Z",
    "log.attributes.app": "server",
    "log.attributes.foo": "bar",
    "resource.attributes.host@name": "mydevhost"
}

Expected behavior
I am expecting a new field add_entry_dot_test to be added based on the condition add_when: /log.attributes.foo == "bar". I am getting new field add_entry_test based on the condition add_when: /severityText == "Info". Looks like data prepper expression-syntax is unable to read fields inside log.attributes or resource.attributes.

Environment (please complete the following information):

  • OS: [Ubuntu 22.04.5 LTS]
  • Docker [27.3.1]
  • Data Prepper [Latest]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.