opensearch-project / opensearch-project/data-prepper

Add AWS credentials for loading S3 files in pipeline.yaml to data-prepper-config.yaml

Open
#2,771 0 comments 1 reaction 1 assignee View on GitHub

@asifsmohammed is already working on this.

Since May 26, 2023.

enhancement
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.
Right now, to load files from S3 in pipelines.yaml in any plugin pipeline author has to provide AWS credentials in each plugin. This is making the pipeline look noisy. Currently we support loading ISM policy(ism_policy_file ) and index template(template_file) from S3.

Describe the solution you'd like
Leverage the extensions work and create an s3_configuration extension in data-prepper-config.yaml.

aws:
  # New configuration
  s3_configurations:
    authentication: my-custom-role
  credentials:
    default:
      sts_role_arn: arn:aws:iam::99123456789:role/OsiDataPlaneRole
      region: us-east-1
    my-custom-role:
      sts_role_arn:
      region:

Now, pipeline author don't have to configure the credentials in pipeline and just use S3 URI path to load files from S3.

log-pipeline:
  s3:
    aws:
      sts_role_arn: arn:aws:iam::123456789012:role/MyRole
      region: us-east-1
  sink:
    - opensearch:
        hosts: [""]
        ism_policy_file: "s3://filepath"

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.