opensearch-project / opensearch-project/data-prepper
Add AWS credentials for loading S3 files in pipeline.yaml to data-prepper-config.yaml
@asifsmohammed is already working on this.
Since May 26, 2023.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
Right now, to load files from S3 in pipelines.yaml in any plugin pipeline author has to provide AWS credentials in each plugin. This is making the pipeline look noisy. Currently we support loading ISM policy(ism_policy_file ) and index template(template_file) from S3.
Describe the solution you'd like
Leverage the extensions work and create an s3_configuration extension in data-prepper-config.yaml.
aws:
# New configuration
s3_configurations:
authentication: my-custom-role
credentials:
default:
sts_role_arn: arn:aws:iam::99123456789:role/OsiDataPlaneRole
region: us-east-1
my-custom-role:
sts_role_arn:
region:
Now, pipeline author don't have to configure the credentials in pipeline and just use S3 URI path to load files from S3.
log-pipeline:
s3:
aws:
sts_role_arn: arn:aws:iam::123456789012:role/MyRole
region: us-east-1
sink:
- opensearch:
hosts: [""]
ism_policy_file: "s3://filepath"
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.