opensearch-project / opensearch-project/data-prepper
[BUG] Kafka source with OAuth cluster authentication does not work without schema registry
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
I'm deploying data-prepper chart in version 0.3.1 from repository https://opensearch-project.github.io/helm-charts/ to deploy opensearch data prepper.
The configuration is following:
...
source:
kafka:
bootstrap_servers:
- <broker ip>:9092
topics:
- name: <topic name>
group_id: opensearch
authentication:
sasl:
oauth:
oauth_sasl_mechanism: "OAUTHBEARER"
oauth_client_id: <client id>
oauth_client_secret: <client secret>
oauth_login_scope: openid
oauth_token_endpoint_url: <token endpoint>
oauth_sasl_login_callback_handler_class: "org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler"
...
The problem is that when using oauth authentication, the deployment fails with
Caused by: java.lang.NullPointerException: Cannot invoke "org.opensearch.dataprepper.plugins.kafka.configuration.SchemaConfig.getBasicAuthCredentialsSource()" because the return value of "org.opensearch.dataprepper.plugins.kafka.source.KafkaSourceConfig.getSchemaConfig()" is null
The source of the problem lies in class org.opensearch.dataprepper.plugins.kafka.util.KafkaSecurityConfigurer on line 207 - 208
if (kafkaClusterAuthConfig instanceof KafkaSourceConfig &&
"USER_INFO".equalsIgnoreCase(((KafkaSourceConfig) kafkaClusterAuthConfig).getSchemaConfig().getBasicAuthCredentialsSource())) {
there is a missing null check on getSchemaConfig() result, causing the configurer to fail when schema configuration is not specified.
I have not found a workaround to this issue, as specifying anything to schema part of the configuration triggers the schema validation.
To Reproduce
Steps to reproduce the behavior:
- Set up Keycloak or other OAuth2 provider
- Set up OpenSearch Data Prepper helm chart with beforementioned configuration, without defining
schemaconfiguration - Deploy the chart
Expected behavior
Data prepper instance starts normally, connects to Kafka instance authenticating using OAuth and starts to consume events from defined topic without any schema registry connection or message schema validation.
Actual behavior
Deployment fails with
Caused by: java.lang.NullPointerException: Cannot invoke "org.opensearch.dataprepper.plugins.kafka.configuration.SchemaConfig.getBasicAuthCredentialsSource()" because the return value of "org.opensearch.dataprepper.plugins.kafka.source.KafkaSourceConfig.getSchemaConfig()" is null
rendering the source unusable in this configuration.
Environment (please complete the following information):
- K8s AKS
- OS Image: CBL-Mariner/Linux
- Operating System: linux
- Architecture: amd64
- Container Runtime Version: containerd://1.6.26
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in data-prepper-plugins/kafka-plugins/src/main/java/org/opensearch/dataprepper/plugins/kafka/util/KafkaSecurityConfigurer.java around lines 207-208, and trace how KafkaSourceConfig handles an absent schema configuration. Reproduce the OAuth-only Kafka source configuration from the issue and verify that the deployment starts without schema registry settings; the relevant Kafka plugin tests are the next place to check for coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka, kubernetes
- Domain
- authentication, data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100