opensearch-project / opensearch-project/data-prepper
Publish a library of modeling and transformation code for various data sources
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
Our team is working to make migrating onto OpenSearch easier for users. Part of this work includes validating the behavioral and performance differences between an original datastore and a prospective OpenSource cluster. We're planning to provide tools for users using a wide array of existing RESTful systems (e.g. solr). While most users will need to change their upstream clients to make OpenSearch API calls, doing that for a POC where they can compare the systems will require a significant and broad investment.
Instead, we're hoping to map incoming requests that were captured for the source cluster and replay them as appropriate (when appropriate) to an OpenSearch cluster. This code can provide value without needing to be production-grade. Some types of messages may have trivial transformations, while others might require more input from the customer. For example, this type of data mapping isn't supported - how should it be handled (ignore/drop the message, uses a different type, etc)? Users of our tools should be able to provide additional mappings to transform messages as they best see fit. From there, they can determine the correctness of their transforms and the OpenSearch cluster. The transformations being done will be illuminating so that users can understand what upstream changes they'll need to make.
Describe the solution you'd like
I expect that DataPrepper already has some level of support for data models from other data stores.
Our needs and the needs of DataPrepper are both aligned on converting data from multiple sources into OpenSearch. They may diverge on the need to do this on a request by request basis, for supporting readonly requests as well, or needing to transform responses. However, we would like to be able to leverage any models or support for transformation that DataPrepper may already have as it may relate to RESTful traffic. Likewise, we'd like to contribute back any code that could be useful for the DataPrepper team or to others.
Extracting any modeling and conversion logic into a separate library or package is one possible way to leverage your work and to centralize a taxonomy of formats that could be useful for others.
Describe alternatives you've considered
We could invoke DataPrepper components directly as a service or through various command line tools. We are still scoping our precise requirements for traffic mirroring for clustering. However, we'll likely a minimal-weight solution that we can exercise significant control over. In some cases, we may be
- Replaying messages through multiple potential target clusters.
- Rerunning requests as user-transformation configuration or logic may change.
- Creating faults (disable a node, fill its disk, etc) to understand how the target systems will perform.
- Run at faster rates to expedite the process and to create additional stresses.
- Replaying billions of messages for validation.
Additional context
We expect that DataPrepper, out of the box, as it has been intended to be used, could be of great value for doing an initial load on a target cluster, which is required to establish the base-case for any of our validation runs.
This request is specifically for how we can benefit from work that you may have done to support other parts of our overall story.
I expect that this issue, if you choose to take it, will require further refinement & discussion. We'd be happy to work with you to find the most appropriate solution!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing DataPrepper components that model or transform data from RESTful sources. Clarify whether the work belongs in a separate library and which request, response, and mapping cases are in scope. Done criteria require a refined design and agreed contribution boundaries, which this issue does not yet define.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100