opensearch-project / opensearch-project/data-prepper

Use OpenSearch constants while still supporting OpenDistro

Open
#877 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

backlog
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.

Data Prepper's opensearch plugin currently supports both OpenSearch 1.0 clusters and OpenDistro Elasticsearch 7.10 clusters. It supports both by using "opendistro" APIs, indices, and property names.

Examples:

All the constants are available by searching for opendistro.

These should be replaced with the new values in OpenSearch.

Describe the solution you'd like

Data Prepper can determine the cluster type. It can do this by making an initial request to the main endpoint (/) and getting the distribution value. Then it uses the correct constants.

We would need to verify if this will work for early versions of OpenSearch (say, 1.0.0). Do these versions all use the new strings?

Describe alternatives you've considered (Optional)

Configuration
A relatively simple solution is adding a property to the opensearch plugin which determines if it supports opendistro or opensearch. Then these usages will vary depending on the cluster type.

Example:

sink:
  opensearch:
    hosts: ["https://localhost:9200"]
    distribution: opendistro

In the 1.x series of Data Prepper, the default should be opendistro since this is the current behavior (it will continue to work with OpenSearch 1.x clusters). But, in Data Prepper 2.x, the default should be opensearch.

Split Plugins

The opensearch plugin could be split into both opensearch and opendistro plugins. This might be necessary in the long-term depending on much these implementations must differ. But, I don't believe it is necessary presently.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with IndexConstants.java, IsmPolicyManagement.java, and the OpenSearchSinkIT.java usages linked in the issue, then search the Java sources for “opendistro”. Determine how cluster distribution should select constants and how early OpenSearch versions behave. Done means the plugin supports the intended OpenSearch and OpenDistro compatibility path with coverage for the affected integrations.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.