opensearch-project / opensearch-project/data-prepper

Supporting OpenSearch sink delete action based on the primary key and operation field in the event data

Open
#5,878 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.
In the case where you have a primary key field exist in your data, but your sink is forcing you to choose auto generated id field ( like Vector collection in OpenSearch), DataPrepper currently does not support deletes operation on the sink because we won't be able to map the _id field while indexing the document.

Requesting this feature to support primary key based document look up and delete operation in this case.

Describe the solution you'd like
In the sink mapping, additional configuration like below can explain the required meta data needed to lookup a document based on the primary key value in the event and apply given action based on the mappings given in the config.

sink:
- OpenSearch
  query_lookup:
    query_when: "getMetadata(\"opensearch_action\") == \"delete\""
    query_term: "pk"
    action_on_found: "delete"
    query_duration: "PT3M"

Additional context
if AOSS sink has replication factor >=2 then each AOSS in AWS gets their auto generated id then this implementation will not be enough but at least it helps in the single AOSS instance case.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the OpenSearch sink and its sink-mapping configuration, then trace how event metadata and primary-key values are handled. Done means the configuration can select delete actions using the event operation and primary key, with behavior covering the stated single-AOSS case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.