opensearch-project / opensearch-project/data-prepper
Support to compare new and old images for DDB streams
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
I have a use case where I need to create/update multiple documents in OpenSearch for a single DDB item. The item basically contains a map out of which only some of entries generally get updated. Right now I need to generate all the documents for the item irrespective of the particular map entry that got updated. This is pretty inefficient and results in multiple unnecessary update calls to OpenSearch.
Describe the solution you'd like
In case NEW_AND_OLD_IMAGES stream view is enabled in DDB, there should be an option in OSI, to extract both the images separately. This will allow me to compare the new and old images and find out the impacted entries and only update those in OpenSearch. This feature should allow us to extract the both the images in aws_lambda processor.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the aws_lambda processor's handling of DynamoDB NEW_AND_OLD_IMAGES and inspect its existing processor tests. Determine how the processor can expose both images separately so callers can compare them. Done means the new option provides distinct old and new images for downstream processing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100