opensearch-project / opensearch-project/data-prepper
Validate data prepper configurations without running Data Prepper
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 355
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
As a user of data prepper, I would like to be able to validate my pipeline configurations without having to start the entirety of data prepper with multiple servers and without having to use real permissions for opensearch, s3, etc.
Describe the solution you'd like
A new module within data prepper called data-prepper-validation-api. This module would provide a library to validate pipeline configuration without starting the entirety of data prepper. It will utilize data-prepper-core code to convert the pipeline configuration into the model for each data prepper plugin, and run the jsr380 validations for those plugins. Additionally, this module would be responsible for constructing instances of plugins with a dependency on only the configuration that the plugin uses.
In order to achieve this, the data prepper directory structure will need to change to the structure proposed in https://github.com/opensearch-project/data-prepper/issues/1503 for data-prepper-core.
After these directory structure changes are complete (only need the changes for splitting out data-prepper-core), we will add the data-prepper-validation-api
The data-prepper-validation-api module will take a dependency on all the data-prepper-plugins that are configured in the pipeline configuration, as well as some of the libraries extracted from data-prepper-core (data-prepper-pipeline, data-prepper-plugin-framework), and will provide a library to run these validations given a pipeline configuration yaml string, and return error messages for invalid configurations. The following dependency hierarchy will be the end result
data-prepper-core
+---- data-prepper-validations
+---- data-prepper-plugin-framework
+---- data-prepper-pipeline
data-prepper-validations
+---- data-prepper-pipeline
+---- data-prepper-plugin-framework
data-prepper-validation-api
+---- data-prepper-validations
+---- opensearch
+---- s3-source
+---- grok-processor
... This can have all plugins so that it can perform actual validations
In order to validate more than just the jsr380 validations, plugins will need to be instantiated just with the configuration model associated with that plugin. Additionally, some plugins do not use jsr380 and the @DataPrepperPluginConstructor annotation for their configurations. For example, to run validations that the grok patterns configured in a grok processor are valid, the grok processor will need to be instantiated because it takes a PluginSetting object in its constructor and will validate that PluginSetting itself, rather than having data-prepper-core validate it.
While we could start with just validating plugin models with jsr380, I am proposing that we add an optional annotation to data-prepper-api that can be used by all plugins, that being @DataPrepperValidateApi. This annotation could be added to either an existing or new constructor that only requires the configuration, whether is it a PluginSetting or a custom config that is converted by the data-prepper-plugin-framework. The data-prepper-validation-api would then look for this annotation on plugins (if it is not found then no extra validations are run), and use it to instantiate the plugin with its configuration. The plugin would then be able to run any validations that it reasonably can in this constructor without creating all of the additional dependencies (servers, clients, etc.), and would be able to provide error messages for this. For example, the grok processor could have the following constructor,
@DataPrepperValidateApi
GrokProcessor(final PluginSetting pluginSetting, final List<String> errors) {
final GrokProcessorConfig grokConfig = buildConfig(pluginSetting);
errors.add(validateGrokPatterns());
}
Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Data Prepper directory structure changes proposal (https://github.com/opensearch-project/data-prepper/issues/1503)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the data-prepper-core directory restructuring proposed in issue 1503, then trace the existing data-prepper-core code used to convert pipeline configuration into plugin models. Define the data-prepper-validation-api module and its dependency hierarchy, including plugin configuration validation and optional annotated constructors. Done means a library can accept a pipeline configuration YAML string and return validation errors without starting Data Prepper or requiring external permissions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100