Have the option to write Event Metadata for failed documents to the DLQ
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
Research direction
No files or tests are named in the issue. Start by locating the DLQ configuration and document serialization entry points, then trace how EventMetadata is available for failed events. Done means a false-by-default dlq.include_event_metadata option and an eventMetadata field in DLQ documents when enabled, with coverage for both settings.
Written by the indexing model from the issue text.
Description
Is your feature request related to a problem? Please describe.
As a user of Data Prepper's DLQ, I would like to also get the metadata of Events in the case that I was using those values in my pipeline for the failed DLQ data. This would allow me to replay DLQ data in the exact same way as the previous pipeline attempted.
Describe the solution you'd like
A configurable option at the dlq level for include_event_metadata, which would default to false.
dlq:
include_event_metadata: true
s3:
bucket: "dlq-bucket-ddb"
key_path_prefix: "osi-test-dlq-pipeline/logs/dlq/%{yyyy}/%{MM}/%{dd}"
region: "us-west-2"
sts_role_arn: "arn:aws:iam::870201406020:role/osis-to-domain-ingestion-role"
When set to true, the DLQ documents will contain an extra field for eventMetadata, which contains the EventMetadata map.
Describe alternatives you've considered (Optional)
Default all DLQ documents to contain the Event metadata
Additional context
Add any other context or screenshots about the feature request here.
- Dominant language
- Java
- Stars
- 374
- Forks
- 355
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from opensearch-project/data-prepper
-
[BUG] Kafka sink throws NullPointerException on every shutdown when thread_wait_time is not set Openbug untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
opensearch-project/data-prepper#7130 ·
-
untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
opensearch-project/data-prepper#7112 ·
-
Mend: dependency security vulnerability
Difficulty 1/5 Under an hour Newbie friendliness 72/100
opensearch-project/data-prepper#7008 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 65/100
opensearch-project/data-prepper#6976 · 1 comment ·
-
Mend: dependency security vulnerability
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
opensearch-project/data-prepper#6919 ·
All issues in opensearch-project/data-prepper
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
bug needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 1/5 Under an hour Newbie friendliness 94/100
objectionary/hone-maven-plugin#1061 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
spring-projects/spring-modulith#1895 ·