opensearch-project / opensearch-project/data-prepper

Provide a configuration for OpenSearch Data Streams

Open
#6,828 0 comments 2 reactions 1 assignee View on GitHub

@dlvenable is already working on this.

Since May 6, 2026.

enhancement
Dominant language
Java
Stars
374
Forks
355
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.

In #6249, we added support for the OpenSearch sink to automatically detect if an index is a data stream.

In some cases, users are seeing inconsistencies. This could be because the permissions to detect are not present, or if the index is not always a data stream (perhaps on other clusters)

Describe the solution you'd like

Provide a new sink configuration to control data streams named data_stream.

The possible values are: automatic (the default), always, never.

opensearch:
    index: "my-index"
    data_stream: "automatic"  # or "always" / "never"

The enum would look something like this:

public enum DataStreamStatus {
      AUTOMATIC("automatic"),
      ALWAYS("always"),
      NEVER("never");
  }

Describe alternatives you've considered (Optional)

N/A

Additional context

Issues: #2037, #2038
PR: #6249

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.