opensearch-project / opensearch-project/data-prepper

[BUG] In OpenSearch pipelines, ${pipelineName} variable does not get replaced in s3 key_path_prefix

Open
#3,609 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug

The ${pipelineName} variable does not get replaced in this field:

  sink:
    - opensearch:
        ...
        dlq:
          s3:
            bucket: "our_dlq_bucket"
            key_path_prefix: "${pipelineName}/logs/dlq"

To Reproduce
Steps to reproduce the behavior:
Create a pipeline with a DLQ for the sink and use the ${pipelineName} variable to construct an object key prefix.

Object keys will contain "${pipelineName}" instead of the pipeline name.

Expected behavior
Object key should contain the actual pipeline name

Screenshots
n/a

Environment (please complete the following information):
n/a

Additional context
This is trivial to hard-code the pipeline name in each pipeline definition, but most users will expect this to work like it does for the ingestion path: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/creating-pipeline.html#pipeline-path

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source files or tests are named. Start by locating the DLQ S3 key_path_prefix handling and compare it with the ingestion path referenced in the issue. Confirm that ${pipelineName} resolves to the pipeline name in generated object keys and add coverage for the reproduced configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.