opensearch-project / opensearch-project/data-prepper
[BUG] add_entries processor add_when is not working when checking for fields within log.attributes or resource.attributes
@san81 is already working on this.
Since Jan 7, 2025.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
The add_entries processor add_when is not working when checking for fields inside log.attributes or resource.attributes. It is working work when I check for a top level field such as severityText
I just setup data prepper docker container as mentioned in here
docker run --name data-prepper -p 2021:2021 -v /${PWD}/pipelines.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml opensearchproject/data-prepper:latest
This is my data prepper config:
otel-opensearch-pipeline:
workers: 1
delay: "5000"
source:
otel_logs_source:
ssl: false
port: 2021
#compression: gzip
processor:
- add_entries:
entries:
- key: "add_entry_test"
value: "done"
add_when: /severityText == "Info"
- key: "add_entry_dot_test"
value: done
add_when: /log.attributes.foo == "bar"
- date:
from_time_received: true
destination: "@timestamp"
sink:
- stdout:
Also in the same machine I have setup opentelemetry-collector-contrib:0.116.1 to receive OTEL logs and forward to data prepper
docker run -p 4317:4317 -v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml otel/opentelemetry-collector-contrib:0.116.1
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
processors:
batch:
exporters:
otlp:
endpoint: "myhost:2021"
tls:
insecure: true
service:
pipelines:
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlp]
I am using Telemetry generator for OpenTelemetry to generate OTEL logs
./telemetrygen logs --body "2024-12-12:00:00:00 INFO This is a test message" --otlp-attributes host.name=\"mydevhost\" --telemetry-attributes foo=\"bar\" --trace-id ae87dadd90e9935a4bc9660628efd569 --span-id 5828fa4960140870 --duration 1s --otlp-insecure
OUTPUT
{
"traceId": "ae87dadd90e9935a4bc9660628efd569",
"spanId": "5828fa4960140870",
"severityText": "Info",
"flags": 0,
"time": "2024-12-27T10:01:38.195840064Z",
"severityNumber": 9,
"droppedAttributesCount": 0,
"serviceName": null,
"body": "2024-12-12:00:00:00 INFO This is a test message",
"observedTime": "1970-01-01T00:00:00Z",
"schemaUrl": "https://opentelemetry.io/schemas/1.4.0",
"add_entry_test": "done",
"@timestamp": "2024-12-27T10:01:38.330Z",
"log.attributes.app": "server",
"log.attributes.foo": "bar",
"resource.attributes.host@name": "mydevhost"
}
Expected behavior
I am expecting a new field add_entry_dot_test to be added based on the condition add_when: /log.attributes.foo == "bar". I am getting new field add_entry_test based on the condition add_when: /severityText == "Info". Looks like data prepper expression-syntax is unable to read fields inside log.attributes or resource.attributes.
Environment (please complete the following information):
- OS: [Ubuntu 22.04.5 LTS]
- Docker [27.3.1]
- Data Prepper [Latest]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.