opensearch-project / opensearch-project/data-prepper
[BUG] Kafka sink throws NullPointerException on every shutdown when thread_wait_time is not set
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
KafkaSinkConfig.threadWaitTime is a Long with no default value, and
KafkaSink.shutdown() unboxes it into
ExecutorService.awaitTermination(long, TimeUnit):
@JsonProperty("thread_wait_time")
private Long threadWaitTime; // KafkaSinkConfig.java
private long calculateLongestThreadWaitingTime() {
return kafkaSinkConfig.getThreadWaitTime(); // KafkaSink.java — NPE when unset
}
thread_wait_time is not marked @NotNull, is not documented as required, and
has no default, so any pipeline that omits it throws NullPointerException on
every shutdown. The exception surfaces on a sink worker thread rather than
the pipeline thread, so the pipeline never shuts down cleanly and ingestion
stops with nothing in the log naming a cause.
To Reproduce
Steps to reproduce the behavior:
- Configure any pipeline with a
kafkasink and omitthread_wait_time:sink: - kafka: bootstrap_servers: ["kafka-server:9092"] encryption: type: none serde_format: json topic: name: my-topic - Start Data Prepper and let the pipeline process at least one batch.
- Stop Data Prepper (
SIGTERM, or stop the container). - See the error in the log:
Exception in thread "linux-syslog-messages-sink-worker-26-thread-1" java.lang.NullPointerException: Cannot invoke "java.lang.Long.longValue()" because the return value of KafkaSinkConfig.getThreadWaitTime() is null at KafkaSink.calculateLongestThreadWaitingTime(KafkaSink.java:194) at KafkaSink.shutdown(KafkaSink.java:178)
Expected behavior
thread_wait_time is optional, so omitting it should apply a documented default
and shut the sink down cleanly. Adding thread_wait_time: 5000 confirms this is
the only thing missing — the NPE is replaced by
KafkaSink - Producer shutdown successfully....
Screenshots
N/A — log output inline above.
Environment (please complete the following information):
- OS: Ubuntu 24.04
- Version: OpenSearch Data Prepper 2.16.0 (
docker.io/opensearchproject/data-prepper:2.16.0), running under podman. Also verified present onmainat commit7222e0d0f(2.17.0-SNAPSHOT). - Kafka:
docker.io/apache/kafka:4.3.1, KRaft, PLAINTEXT listener
Additional context
Two possible fixes, and we have applied both locally:
- A default on the config field —
private Long threadWaitTime = DEFAULT_THREAD_WAIT_TIME_MS;with
static final long DEFAULT_THREAD_WAIT_TIME_MS = 1000L. 1000 ms is the value
already used throughout the existing sink tests and
sample-pipelines-sink.yaml. - A null guard at the unboxing site, so an explicit
thread_wait_time: null
cannot reintroduce the crash.
Happy to open a PR with both plus unit tests. Note that a restart timer is not a
usable workaround: with this NPE present, shutting down all pipelines took over
11 minutes in our deployment (Workers did not terminate in PT30S, forcing termination, repeated per pipeline).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with KafkaSinkConfig.java and KafkaSink.java, especially getThreadWaitTime(), calculateLongestThreadWaitingTime(), and shutdown(). Review the existing sink tests and sample-pipelines-sink.yaml for the established 1000 ms value. Done means an omitted or explicitly null thread_wait_time no longer causes shutdown to throw, with tests covering the configuration and clean shutdown behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- backend, stream-processing
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100