opensearch-project / opensearch-project/data-prepper
[BUG] dlq object should include the dynamic index runtime value instead of the literal value in pipeline configuration
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
A clear and concise description of what the bug is.
given the following snippet in pipeline YAML:
opensearch:
index: "${/index}"
dlq:
s3:
bucket: ...
The dlq object will end up having the following key-value:
"failedData":{"index":"${/index}"
in which /failedData/index value is the literal value in the pipeline config.
To Reproduce
Steps to reproduce the behavior:
- Go to '...'
- Click on '....'
- Scroll down to '....'
- See error
Expected behavior
The expected behavior should be the runtime value extracted from individual event:
"failedData":{"index":"<<runtime value from individual event under /index json path>>"
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
- OS: [e.g. Ubuntu 20.04 LTS]
- Version [e.g. 22]
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue with the pipeline YAML shown, using a dynamic /index value in an individual event and inspecting the resulting failedData/index value in the DLQ object. Trace the pipeline's DLQ handling from the opensearch configuration and verify that done means failedData/index contains the event's runtime value rather than the literal "${/index}".
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100