opensearch-project / opensearch-project/data-prepper

[BUG] dlq object should include the dynamic index runtime value instead of the literal value in pipeline configuration

Open
#4,808 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
A clear and concise description of what the bug is.

given the following snippet in pipeline YAML:

opensearch:
    index: "${/index}"
    dlq:
          s3:
            bucket: ...

The dlq object will end up having the following key-value:

"failedData":{"index":"${/index}"

in which /failedData/index value is the literal value in the pipeline config.

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
The expected behavior should be the runtime value extracted from individual event:

"failedData":{"index":"<<runtime value from individual event under /index json path>>"

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

  • OS: [e.g. Ubuntu 20.04 LTS]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue with the pipeline YAML shown, using a dynamic /index value in an individual event and inspecting the resulting failedData/index value in the DLQ object. Trace the pipeline's DLQ handling from the opensearch configuration and verify that done means failedData/index contains the event's runtime value rather than the literal "${/index}".

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.