opensearch-project / opensearch-project/data-prepper
[BUG] User agent processor gives java.lang.NullPointerException if entry does not contain source field
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
With Data Prepper 2.12.1 installed as Docker container I get java.lang.NullPointerException when using the User agent processor if entry does contain defined source field.
To Reproduce
Steps to reproduce the behavior:
- Have a pipeline like this configured
source:
http:
port: 2021
ssl: false
health_check_service: true
processor:
- user_agent:
source: agent
target: parsed_agent
sink:
- opensearch:
hosts:
- https://HOST:PORT
- https://HOST:PORT
- https://HOST:PORT
insecure: true
username: USER
password: PASS
index: TEST
- Ingest data which may or may not contain field
agent - Following errors get logged (for each ingested entry without field
agentI believe)
Reading pipelines and data-prepper configuration files from Data Prepper home directory.
/usr/bin/java
Found openjdk version of 17.0
2025-08-22T15:33:33,583 [main] INFO org.opensearch.dataprepper.pipeline.parser.transformer.DynamicConfigTransformer - No transformation needed
2025-08-22T15:33:35,782 [main] INFO org.opensearch.dataprepper.plugins.kafka.extension.KafkaClusterConfigExtension - Applying Kafka Cluster Config Extension.
2025-08-22T15:33:37,104 [main] WARN org.opensearch.dataprepper.plugins.source.loghttp.HTTPSource - Creating http source without authentication. This is not secure.
2025-08-22T15:33:37,105 [main] WARN org.opensearch.dataprepper.plugins.source.loghttp.HTTPSource - In order to set up Http Basic authentication for the http source, go here: https://github.com/opensearch-project/data-prepper/tree/main/data-prepper
-plugins/http-source#authentication-configurations
2025-08-22T15:33:37,798 [main] INFO org.opensearch.dataprepper.plugins.geoip.extension.GeoIPDatabaseManager - Downloading GeoIP database to /usr/share/data-prepper/data/geoip/blue_database
2025-08-22T15:33:44,367 [main] WARN org.opensearch.dataprepper.core.pipeline.server.config.DataPrepperServerConfiguration - Creating data prepper server without authentication. This is not secure.
2025-08-22T15:33:44,372 [main] WARN org.opensearch.dataprepper.core.pipeline.server.config.DataPrepperServerConfiguration - In order to set up Http Basic authentication for the data prepper server, go here: https://github.com/opensearch-project/da
ta-prepper/blob/main/docs/core_apis.md#authentication
2025-08-22T15:33:44,831 [log-ingest-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
2025-08-22T15:33:44,848 [main] WARN org.opensearch.dataprepper.core.pipeline.server.HttpServerProvider - Creating Data Prepper server without TLS. This is not secure.
2025-08-22T15:33:44,852 [main] WARN org.opensearch.dataprepper.core.pipeline.server.HttpServerProvider - In order to set up TLS for the Data Prepper server, go here: https://github.com/opensearch-project/data-prepper/blob/main/docs/configuration.m
d#server-configuration
2025-08-22T15:33:44,867 [log-ingest-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
2025-08-22T15:33:44,907 [log-ingest-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the trust all strategy
2025-08-22T15:33:45,480 [log-ingest-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initialized OpenSearch sink
2025-08-22T15:33:45,801 [log-ingest-pipeline-sink-worker-2-thread-1] WARN com.linecorp.armeria.common.CommonPools - Failed to register the common worker group as non-blocking for Reactor. Please consider upgrading Reactor to 3.7.0 or newer.
2025-08-22T15:33:45,948 [log-ingest-pipeline-sink-worker-2-thread-1] WARN org.opensearch.dataprepper.plugins.server.CreateServer - Creating http without SSL/TLS. This is not secure.
2025-08-22T15:33:45,949 [log-ingest-pipeline-sink-worker-2-thread-1] WARN org.opensearch.dataprepper.plugins.server.CreateServer - In order to set up TLS for the http, go here: https://github.com/opensearch-project/data-prepper/tree/main/data-prep
per-plugins/http-source#ssl
2025-08-22T15:33:46,017 [log-ingest-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.server.CreateServer - HTTP source health check is enabled
2025-08-22T15:33:46,545 [log-ingest-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.loghttp.HTTPSource - Started http source on port 2021...
2025-08-22T15:33:50,241 [log-ingest-pipeline-processor-worker-1-thread-1] ERROR org.opensearch.dataprepper.plugins.processor.useragent.UserAgentProcessor - An exception occurred when parsing user agent data from event [org.opensearch.dataprepper.model.log.JacksonLog@79ad1a39] with source key [agent]
java.lang.NullPointerException: null
at java.base/java.util.Objects.requireNonNull(Objects.java:209) ~[?:?]
at org.opensearch.dataprepper.plugins.processor.useragent.UserAgentProcessor.doExecute(UserAgentProcessor.java:57) ~[data-prepper-user-agent-processor-2.12.1.jar:?]
at org.opensearch.dataprepper.model.processor.AbstractProcessor.lambda$execute$0(AbstractProcessor.java:54) ~[data-prepper-api-2.12.1.jar:?]
at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:69) [micrometer-core-1.14.4.jar:1.14.4]
at org.opensearch.dataprepper.model.processor.AbstractProcessor.execute(AbstractProcessor.java:54) [data-prepper-api-2.12.1.jar:?]
at org.opensearch.dataprepper.core.pipeline.PipelineRunnerImpl.runProcessorsAndProcessAcknowledgements(PipelineRunnerImpl.java:105) [data-prepper-core-2.12.1.jar:?]
at org.opensearch.dataprepper.core.pipeline.PipelineRunnerImpl.runAllProcessorsAndPublishToSinks(PipelineRunnerImpl.java:55) [data-prepper-core-2.12.1.jar:?]
at org.opensearch.dataprepper.core.pipeline.ProcessWorker.doRun(ProcessWorker.java:80) [data-prepper-core-2.12.1.jar:?]
at org.opensearch.dataprepper.core.pipeline.ProcessWorker.run(ProcessWorker.java:40) [data-prepper-core-2.12.1.jar:?]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
2025-08-22T15:33:50,251 [log-ingest-pipeline-processor-worker-1-thread-1] ERROR org.opensearch.dataprepper.plugins.processor.useragent.UserAgentProcessor - An exception occurred when parsing user agent data from event [org.opensearch.dataprepper.model.log.JacksonLog@54ba2d01] with source key [agent]
java.lang.NullPointerException: null
at java.base/java.util.Objects.requireNonNull(Objects.java:209) ~[?:?]
at org.opensearch.dataprepper.plugins.processor.useragent.UserAgentProcessor.doExecute(UserAgentProcessor.java:57) ~[data-prepper-user-agent-processor-2.12.1.jar:?]
at org.opensearch.dataprepper.model.processor.AbstractProcessor.lambda$execute$0(AbstractProcessor.java:54) ~[data-prepper-api-2.12.1.jar:?]
at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:69) [micrometer-core-1.14.4.jar:1.14.4]
at org.opensearch.dataprepper.model.processor.AbstractProcessor.execute(AbstractProcessor.java:54) [data-prepper-api-2.12.1.jar:?]
at org.opensearch.dataprepper.core.pipeline.PipelineRunnerImpl.runProcessorsAndProcessAcknowledgements(PipelineRunnerImpl.java:105) [data-prepper-core-2.12.1.jar:?]
at org.opensearch.dataprepper.core.pipeline.PipelineRunnerImpl.runAllProcessorsAndPublishToSinks(PipelineRunnerImpl.java:55) [data-prepper-core-2.12.1.jar:?]
at org.opensearch.dataprepper.core.pipeline.ProcessWorker.doRun(ProcessWorker.java:80) [data-prepper-core-2.12.1.jar:?]
at org.opensearch.dataprepper.core.pipeline.ProcessWorker.run(ProcessWorker.java:40) [data-prepper-core-2.12.1.jar:?]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Environment (please complete the following information):
- OS: Ubuntu 24.04 LTS
- Version: Data Prepper 2.12.1
Additional context
In my use case I ingest log entries which may or may not contain user agent. I could not figure out any way to pass only entries containing agent information to the User agent processor so I tought I could pass all entries to it without getting erros.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at UserAgentProcessor.java line 57, identified in the stack trace, and reproduce the processor with an event that lacks the configured agent field. Determine the expected handling for missing user-agent data from the issue context, then verify that such entries no longer produce a NullPointerException while entries containing agent data continue to be processed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100