opensearch-project / opensearch-project/data-prepper
[BUG] Index prefix parsing bug in OpenSearch sink
@oeyh is already working on this.
Since May 14, 2024.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
Index prefix is not parsed correctly when the index name contains dynamic date pattern and the date pattern is not prefixed or suffixed with a hypen, for example, if index is configured as index-prefix.%{yyyy-MM-dd}, the parsed index prefix will be index-prefix.%{yyyy-MM-dd} instead of index-prefix.
This bug can manifest itself as a misleading [security_exception] authentication/authorization failure when pipeline tries to check if a corresponding index template exists. See details below.
To Reproduce
Steps to reproduce the behavior:
- Configure a pipeline with this opensearch sink config pointing to a public AOSS collection:
sink:
- opensearch:
hosts:
- https://xxxx.us-east-1.aoss.amazonaws.com
index_type: custom
index: "prefix.%{yyyy-MM-dd}"
aws:
sts_role_arn: arn:aws:iam::xxx:role/OpenSearchServerlessPipelineRole
region: us-east-1
serverless: true
template_type: index-template
template_content: '{"template":{"mappings":{"date_detection":false}}}'
- Start the pipeline, pipeline fails to init sink with this error message:
org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [security_exception] authentication/authorization failure
at org.opensearch.client.transport.aws.AwsSdk2Transport.parseResponse(AwsSdk2Transport.java:473) ~[opensearch-java-2.8.1.jar:?]
at org.opensearch.client.transport.aws.AwsSdk2Transport.executeSync(AwsSdk2Transport.java:392) ~[opensearch-java-2.8.1.jar:?]
at org.opensearch.client.transport.aws.AwsSdk2Transport.performRequest(AwsSdk2Transport.java:192) ~[opensearch-java-2.8.1.jar:?]
at org.opensearch.client.opensearch.indices.OpenSearchIndicesClient.existsIndexTemplate(OpenSearchIndicesClient.java:571) ~[opensearch-java-2.8.1.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.index.ComposableTemplateAPIWrapper.getTemplate(ComposableTemplateAPIWrapper.java:45) ~[opensearch-2.7.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.index.ComposableIndexTemplateStrategy.getExistingTemplateVersion(ComposableIndexTemplateStrategy.java:28) ~[opensearch-2.7.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.shouldCreateTemplate(AbstractIndexManager.java:292) ~[opensearch-2.7.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.checkAndCreateIndexTemplate(AbstractIndexManager.java:246) ~[opensearch-2.7.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.checkAndCreateIndexTemplate(AbstractIndexManager.java:234) ~[opensearch-2.7.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.setupIndex(AbstractIndexManager.java:224) ~[opensearch-2.7.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.doInitializeInternal(OpenSearchSink.java:235) ~[opensearch-2.7.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.doInitialize(OpenSearchSink.java:193) ~[opensearch-2.7.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.model.sink.AbstractSink.initialize(AbstractSink.java:52) ~[data-prepper-api-2.7.0-SNAPSHOT.jar:?]
...
Expected behavior
Sink should initialize correctly.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
- OS: [e.g. Ubuntu 20.04 LTS]
- Version [e.g. 22]
Additional context
Relevant code:
https://github.com/opensearch-project/data-prepper/blob/1e5e0d06b825923ccce47689d3cbf5bccd365a29/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/AbstractIndexManager.java#L127-L129
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.