opensearch-project / opensearch-project/data-prepper

[BUG] User agent processor gives java.lang.NullPointerException if entry does not contain source field

Open
#6,010 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug good first issue
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
With Data Prepper 2.12.1 installed as Docker container I get java.lang.NullPointerException when using the User agent processor if entry does contain defined source field.

To Reproduce
Steps to reproduce the behavior:

  1. Have a pipeline like this configured
  source:
    http:
      port: 2021
      ssl: false
      health_check_service: true

  processor:
    - user_agent:
         source: agent
         target: parsed_agent

  sink:
    - opensearch:
        hosts:
          - https://HOST:PORT
          - https://HOST:PORT
          - https://HOST:PORT
        insecure: true
        username: USER
        password: PASS
        index: TEST
  1. Ingest data which may or may not contain field agent
  2. Following errors get logged (for each ingested entry without field agent I believe)
Reading pipelines and data-prepper configuration files from Data Prepper home directory.
/usr/bin/java
Found openjdk version  of 17.0
2025-08-22T15:33:33,583 [main] INFO  org.opensearch.dataprepper.pipeline.parser.transformer.DynamicConfigTransformer - No transformation needed
2025-08-22T15:33:35,782 [main] INFO  org.opensearch.dataprepper.plugins.kafka.extension.KafkaClusterConfigExtension - Applying Kafka Cluster Config Extension.
2025-08-22T15:33:37,104 [main] WARN  org.opensearch.dataprepper.plugins.source.loghttp.HTTPSource - Creating http source without authentication. This is not secure.
2025-08-22T15:33:37,105 [main] WARN  org.opensearch.dataprepper.plugins.source.loghttp.HTTPSource - In order to set up Http Basic authentication for the http source, go here: https://github.com/opensearch-project/data-prepper/tree/main/data-prepper
-plugins/http-source#authentication-configurations
2025-08-22T15:33:37,798 [main] INFO  org.opensearch.dataprepper.plugins.geoip.extension.GeoIPDatabaseManager - Downloading GeoIP database to /usr/share/data-prepper/data/geoip/blue_database
2025-08-22T15:33:44,367 [main] WARN  org.opensearch.dataprepper.core.pipeline.server.config.DataPrepperServerConfiguration - Creating data prepper server without authentication. This is not secure.
2025-08-22T15:33:44,372 [main] WARN  org.opensearch.dataprepper.core.pipeline.server.config.DataPrepperServerConfiguration - In order to set up Http Basic authentication for the data prepper server, go here: https://github.com/opensearch-project/da
ta-prepper/blob/main/docs/core_apis.md#authentication
2025-08-22T15:33:44,831 [log-ingest-pipeline-sink-worker-2-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
2025-08-22T15:33:44,848 [main] WARN  org.opensearch.dataprepper.core.pipeline.server.HttpServerProvider - Creating Data Prepper server without TLS. This is not secure.
2025-08-22T15:33:44,852 [main] WARN  org.opensearch.dataprepper.core.pipeline.server.HttpServerProvider - In order to set up TLS for the Data Prepper server, go here: https://github.com/opensearch-project/data-prepper/blob/main/docs/configuration.m
d#server-configuration
2025-08-22T15:33:44,867 [log-ingest-pipeline-sink-worker-2-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
2025-08-22T15:33:44,907 [log-ingest-pipeline-sink-worker-2-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the trust all strategy
2025-08-22T15:33:45,480 [log-ingest-pipeline-sink-worker-2-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initialized OpenSearch sink
2025-08-22T15:33:45,801 [log-ingest-pipeline-sink-worker-2-thread-1] WARN   com.linecorp.armeria.common.CommonPools - Failed to register the common worker group as non-blocking for Reactor. Please consider upgrading Reactor to 3.7.0 or newer.
2025-08-22T15:33:45,948 [log-ingest-pipeline-sink-worker-2-thread-1] WARN  org.opensearch.dataprepper.plugins.server.CreateServer - Creating http without SSL/TLS. This is not secure.
2025-08-22T15:33:45,949 [log-ingest-pipeline-sink-worker-2-thread-1] WARN  org.opensearch.dataprepper.plugins.server.CreateServer - In order to set up TLS for the http, go here: https://github.com/opensearch-project/data-prepper/tree/main/data-prep
per-plugins/http-source#ssl
2025-08-22T15:33:46,017 [log-ingest-pipeline-sink-worker-2-thread-1] INFO  org.opensearch.dataprepper.plugins.server.CreateServer - HTTP source health check is enabled
2025-08-22T15:33:46,545 [log-ingest-pipeline-sink-worker-2-thread-1] INFO  org.opensearch.dataprepper.plugins.source.loghttp.HTTPSource - Started http source on port 2021...
2025-08-22T15:33:50,241 [log-ingest-pipeline-processor-worker-1-thread-1] ERROR org.opensearch.dataprepper.plugins.processor.useragent.UserAgentProcessor - An exception occurred when parsing user agent data from event [org.opensearch.dataprepper.model.log.JacksonLog@79ad1a39] with source key [agent]
java.lang.NullPointerException: null
        at java.base/java.util.Objects.requireNonNull(Objects.java:209) ~[?:?]
        at org.opensearch.dataprepper.plugins.processor.useragent.UserAgentProcessor.doExecute(UserAgentProcessor.java:57) ~[data-prepper-user-agent-processor-2.12.1.jar:?]
        at org.opensearch.dataprepper.model.processor.AbstractProcessor.lambda$execute$0(AbstractProcessor.java:54) ~[data-prepper-api-2.12.1.jar:?]
        at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:69) [micrometer-core-1.14.4.jar:1.14.4]
        at org.opensearch.dataprepper.model.processor.AbstractProcessor.execute(AbstractProcessor.java:54) [data-prepper-api-2.12.1.jar:?]
        at org.opensearch.dataprepper.core.pipeline.PipelineRunnerImpl.runProcessorsAndProcessAcknowledgements(PipelineRunnerImpl.java:105) [data-prepper-core-2.12.1.jar:?]
        at org.opensearch.dataprepper.core.pipeline.PipelineRunnerImpl.runAllProcessorsAndPublishToSinks(PipelineRunnerImpl.java:55) [data-prepper-core-2.12.1.jar:?]
        at org.opensearch.dataprepper.core.pipeline.ProcessWorker.doRun(ProcessWorker.java:80) [data-prepper-core-2.12.1.jar:?]
        at org.opensearch.dataprepper.core.pipeline.ProcessWorker.run(ProcessWorker.java:40) [data-prepper-core-2.12.1.jar:?]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
2025-08-22T15:33:50,251 [log-ingest-pipeline-processor-worker-1-thread-1] ERROR org.opensearch.dataprepper.plugins.processor.useragent.UserAgentProcessor - An exception occurred when parsing user agent data from event [org.opensearch.dataprepper.model.log.JacksonLog@54ba2d01] with source key [agent]
java.lang.NullPointerException: null
        at java.base/java.util.Objects.requireNonNull(Objects.java:209) ~[?:?]
        at org.opensearch.dataprepper.plugins.processor.useragent.UserAgentProcessor.doExecute(UserAgentProcessor.java:57) ~[data-prepper-user-agent-processor-2.12.1.jar:?]
        at org.opensearch.dataprepper.model.processor.AbstractProcessor.lambda$execute$0(AbstractProcessor.java:54) ~[data-prepper-api-2.12.1.jar:?]
        at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:69) [micrometer-core-1.14.4.jar:1.14.4]
        at org.opensearch.dataprepper.model.processor.AbstractProcessor.execute(AbstractProcessor.java:54) [data-prepper-api-2.12.1.jar:?]
        at org.opensearch.dataprepper.core.pipeline.PipelineRunnerImpl.runProcessorsAndProcessAcknowledgements(PipelineRunnerImpl.java:105) [data-prepper-core-2.12.1.jar:?]
        at org.opensearch.dataprepper.core.pipeline.PipelineRunnerImpl.runAllProcessorsAndPublishToSinks(PipelineRunnerImpl.java:55) [data-prepper-core-2.12.1.jar:?]
        at org.opensearch.dataprepper.core.pipeline.ProcessWorker.doRun(ProcessWorker.java:80) [data-prepper-core-2.12.1.jar:?]
        at org.opensearch.dataprepper.core.pipeline.ProcessWorker.run(ProcessWorker.java:40) [data-prepper-core-2.12.1.jar:?]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

Environment (please complete the following information):

  • OS: Ubuntu 24.04 LTS
  • Version: Data Prepper 2.12.1

Additional context
In my use case I ingest log entries which may or may not contain user agent. I could not figure out any way to pass only entries containing agent information to the User agent processor so I tought I could pass all entries to it without getting erros.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at UserAgentProcessor.java line 57, identified in the stack trace, and reproduce the processor with an event that lacks the configured agent field. Determine the expected handling for missing user-agent data from the issue context, then verify that such entries no longer produce a NullPointerException while entries containing agent data continue to be processed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.