opensearch-project / opensearch-project/data-prepper
Supporting OpenSearch sink delete action based on the primary key and operation field in the event data
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
In the case where you have a primary key field exist in your data, but your sink is forcing you to choose auto generated id field ( like Vector collection in OpenSearch), DataPrepper currently does not support deletes operation on the sink because we won't be able to map the _id field while indexing the document.
Requesting this feature to support primary key based document look up and delete operation in this case.
Describe the solution you'd like
In the sink mapping, additional configuration like below can explain the required meta data needed to lookup a document based on the primary key value in the event and apply given action based on the mappings given in the config.
sink:
- OpenSearch
query_lookup:
query_when: "getMetadata(\"opensearch_action\") == \"delete\""
query_term: "pk"
action_on_found: "delete"
query_duration: "PT3M"
Additional context
if AOSS sink has replication factor >=2 then each AOSS in AWS gets their auto generated id then this implementation will not be enough but at least it helps in the single AOSS instance case.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the OpenSearch sink and its sink-mapping configuration, then trace how event metadata and primary-key values are handled. Done means the configuration can select delete actions using the event operation and primary key, with behavior covering the stated single-AOSS case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100