opensearch-project / opensearch-project/data-prepper
Use OpenSearch constants while still supporting OpenDistro
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
Data Prepper's opensearch plugin currently supports both OpenSearch 1.0 clusters and OpenDistro Elasticsearch 7.10 clusters. It supports both by using "opendistro" APIs, indices, and property names.
Examples:
- https://github.com/opensearch-project/data-prepper/blob/06d53ae2329509d99c1c342a4b505935177c70aa/data-prepper-plugins/opensearch/src/main/java/com/amazon/dataprepper/plugins/sink/opensearch/index/IndexConstants.java#L27-L29
- https://github.com/opensearch-project/data-prepper/blob/1183a7a13517885021c023971c77514f654dcc95/data-prepper-plugins/opensearch/src/main/java/com/amazon/dataprepper/plugins/sink/opensearch/index/IsmPolicyManagement.java#L39
- https://github.com/opensearch-project/data-prepper/blob/54f8ce772d4c28468304b670e79bb1b83a13be7d/data-prepper-plugins/opensearch/src/test/java/com/amazon/dataprepper/plugins/sink/opensearch/OpenSearchSinkIT.java#L593 (several usages in this file)
All the constants are available by searching for opendistro.
These should be replaced with the new values in OpenSearch.
Describe the solution you'd like
Data Prepper can determine the cluster type. It can do this by making an initial request to the main endpoint (/) and getting the distribution value. Then it uses the correct constants.
We would need to verify if this will work for early versions of OpenSearch (say, 1.0.0). Do these versions all use the new strings?
Describe alternatives you've considered (Optional)
Configuration
A relatively simple solution is adding a property to the opensearch plugin which determines if it supports opendistro or opensearch. Then these usages will vary depending on the cluster type.
Example:
sink:
opensearch:
hosts: ["https://localhost:9200"]
distribution: opendistro
In the 1.x series of Data Prepper, the default should be opendistro since this is the current behavior (it will continue to work with OpenSearch 1.x clusters). But, in Data Prepper 2.x, the default should be opensearch.
Split Plugins
The opensearch plugin could be split into both opensearch and opendistro plugins. This might be necessary in the long-term depending on much these implementations must differ. But, I don't believe it is necessary presently.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with IndexConstants.java, IsmPolicyManagement.java, and the OpenSearchSinkIT.java usages linked in the issue, then search the Java sources for “opendistro”. Determine how cluster distribution should select constants and how early OpenSearch versions behave. Done means the plugin supports the intended OpenSearch and OpenDistro compatibility path with coverage for the affected integrations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100