opensearch-project / opensearch-project/data-prepper

[BUG] opensearch_api source does not work with Kafka buffer - no ByteDecoder registered

Open
#6,876 0 comments 0 reactions 1 assignee View on GitHub

@divakarsingh is already working on this.

Since May 26, 2026.

enhancement
Dominant language
Java
Stars
374
Forks
355
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
The opensearch_api source does not work when configured with a kafka buffer. Traffic stops flowing — events are written to Kafka but never consumed/reconstructed on the other side. It works correctly with bounded_blocking buffer.

To Reproduce

  1. Configure Data Prepper with opensearch_api source and kafka buffer
  2. Send a _bulk request to Data Prepper
  3. Observe: events are written to Kafka topic but never reach the sink
  4. Switch buffer to bounded_blocking — traffic flows correctly

Expected behavior
The opensearch_api source should register a ByteDecoder that can parse NDJSON bulk format back into Data Prepper events with correct metadata attributes (opensearch_action, opensearch_index, opensearch_id, etc.) when using Kafka buffer.

Screenshots
N/A

Environment (please complete the following information):

  • OS: Linux
  • Version: Data Prepper 2.15.1 / main branch

Additional context
Forum report: https://forum.opensearch.org/t/opensearch-api-source-doesnt-work-with-kafka-buffer/28059

Root cause: When buffer.isByteBuffer() == true (Kafka buffer), the source writes raw bytes via buffer.writeBytes(content.array(), ...). On the consumer side, the buffer needs a ByteDecoder to reconstruct Record<Event> objects from those bytes. OpenSearchAPISource does not override getDecoder(), so no decoder is registered. The Kafka buffer consumer cannot parse the NDJSON bulk format back into events.

The bounded_blocking buffer works because it stores Record<Event> objects directly in a BlockingQueue — no serialization/deserialization needed.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.