opensearch-project / opensearch-project/data-prepper
Add option to shut down and fail data prepper when any sub-pipelines are invalid
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
The default behavior of Data Prepper core is to only shut down when no sub-pipelines are valid. This can lead to a confusing user experience, as someone who wants all sub-pipelines to be running correctly for data prepper to start.
Describe the solution you'd like
An optional parameter in the data-prepper-config.yaml to change this behavior to fail to start Data Prepper when any of the sub-pipelines are not configured properly. This would then throw an exception that stops Data Prepper when any sub-pipelines are misconfigured.
fail_on_any_sub_pipeline_failure: true // defaults to false
Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how data-prepper-config.yaml is parsed and how invalid sub-pipelines affect Data Prepper startup. Determine where the optional fail_on_any_sub_pipeline_failure setting belongs, then follow the existing startup validation tests if present. Done means the default behavior remains unchanged and enabling the option stops startup when any sub-pipeline is invalid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100