opensearch-project / opensearch-project/data-prepper
[BUG] Invalid time pattern placement in dynamic index pattern shuts down pipeline
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
DataPrepper does not allow time patterns to be in dynamic index patterns unless they are the suffix of the pattern: https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/AbstractIndexManager.java#L127
This causes the pipeline to shutdown at runtime when data is sent to a sink who's index pattern violates this condition.
ERROR org.opensearch.dataprepper.pipeline.common.PipelineThreadPoolExecutor - Pipeline [***] process worker encountered a fatal exception, cannot proceed further
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Time pattern can only be a suffix of an index
To Reproduce
Create a pipeline with this index pattern in an opensearch sink:
prefix-%{yyyy.MM.dd}-suffix
Expected behavior
It's unclear why DataPrepper does not support non-suffix time patterns. If this is not a hard restriction then I would expect this validation to be removed. If it is a hard restriction then I would expect this to be validated during initialization rather than throwing an exception and shutting down the pipeline at runtime.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with AbstractIndexManager.java at the validation around line 127 and trace how the OpenSearch sink handles the dynamic index pattern. Determine whether non-suffix time patterns should be supported or rejected during initialization; done means the chosen behavior no longer shuts down the pipeline at runtime for the reproduced pattern.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100