opensearch-project / opensearch-project/data-prepper
Verify plugin conventions as part of the build
@sb2k16 is already working on this.
Since Nov 15, 2024.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
We have a number of inconsistent names in our configurations. And recently we added a plugin name with hyphens instead of underscores (#5138).
Describe the solution you'd like
I'd like to have automation in Data Prepper that verifies that plugin names meet our conventions.
I think this could be done by creating a testing project that has built-in unit tests for scanning for plugins and verifying the naming convention. Additionally, we can have this added automatically to all plugin projects in Data Prepper.
To accomplish this, we could have a new project:
data-prepper-plugin-test
We apply it to all plugins by adding something like the following to data-prepper-plugins/build.gradle
Update the above to the following.
subprojects {
apply plugin: 'data-prepper.publish'
group = 'org.opensearch.dataprepper.plugins'
dependencies {
implementation project(':data-prepper-plugin-test')
}
}
At this point, the data-prepper-plugin-test will be part of every plugin project. Now, we need to have tests in there.
The project data-prepper-plugin-test can take a dependency on data-prepper-plugin-framework so that it can use ClasspathPluginProvider. We would also need to add a new method to ClasspathPluginProvider to get all plugin names found. It might look somewhat like the following.
Set<String> listPlugins() {
if (nameToSupportedTypeToPluginType == null) {
nameToSupportedTypeToPluginType = scanForPlugins();
}
return nameToSupportedTypeToPluginType.keySet();
}
New we can write a unit test that list all plugins and verifies that they meet our naming conventions. Some things we should verify:
- They should be all lowercase
- They should have only
_as a special character
One additional change we'd need is some way to override the conventions. Some of our plugins already have some bad conventions. I propose that plugin authors can add a file to indicate ignores. This can go in src/test/resources/org.opensearch.dataprepper.plugin.test.conventions-configurations.yaml. It might look somewhat like the following:
overrides:
plugin_names:
kinesis-data-streams:
ignore_conventions: [require_underscores]
This would tell the framework to ignore the convention to require underscores for the kinesis-data-streams plugin. We would still perform the other validations on it.
Describe alternatives you've considered (Optional)
Alternative 1: We could create a Gradle plugin in buildSrc which looks for annotations of @DataPrepperPlugin and verifies the name. This would be a good way to verify the plugin names, but would be quite difficult to extend beyond the plugin names to use on the configurations themselves.
Alternative 2: We could try to use Checkstyle to verify the plugin names. This could be quite simple. But, the downside is that it would only cover the @DataPrepperPlugin annotation.
Additional context
Add any other context or screenshots about the feature request here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.