opensearch-project / opensearch-project/data-prepper

[BUG] Kafka source with OAuth cluster authentication does not work without schema registry

Open
#5,883 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
I'm deploying data-prepper chart in version 0.3.1 from repository https://opensearch-project.github.io/helm-charts/ to deploy opensearch data prepper.

The configuration is following:

...
source:
  kafka:
    bootstrap_servers:
      - <broker ip>:9092
    topics:
      - name: <topic name>
        group_id: opensearch
    authentication:
      sasl:
        oauth:
          oauth_sasl_mechanism: "OAUTHBEARER"
          oauth_client_id: <client id>
          oauth_client_secret: <client secret>
          oauth_login_scope: openid
          oauth_token_endpoint_url: <token endpoint>
          oauth_sasl_login_callback_handler_class: "org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler"
...

The problem is that when using oauth authentication, the deployment fails with

Caused by: java.lang.NullPointerException: Cannot invoke "org.opensearch.dataprepper.plugins.kafka.configuration.SchemaConfig.getBasicAuthCredentialsSource()" because the return value of "org.opensearch.dataprepper.plugins.kafka.source.KafkaSourceConfig.getSchemaConfig()" is null

The source of the problem lies in class org.opensearch.dataprepper.plugins.kafka.util.KafkaSecurityConfigurer on line 207 - 208

        if (kafkaClusterAuthConfig instanceof KafkaSourceConfig &&
                "USER_INFO".equalsIgnoreCase(((KafkaSourceConfig) kafkaClusterAuthConfig).getSchemaConfig().getBasicAuthCredentialsSource())) {

there is a missing null check on getSchemaConfig() result, causing the configurer to fail when schema configuration is not specified.

I have not found a workaround to this issue, as specifying anything to schema part of the configuration triggers the schema validation.

To Reproduce
Steps to reproduce the behavior:

  1. Set up Keycloak or other OAuth2 provider
  2. Set up OpenSearch Data Prepper helm chart with beforementioned configuration, without defining schema configuration
  3. Deploy the chart

Expected behavior
Data prepper instance starts normally, connects to Kafka instance authenticating using OAuth and starts to consume events from defined topic without any schema registry connection or message schema validation.

Actual behavior
Deployment fails with

Caused by: java.lang.NullPointerException: Cannot invoke "org.opensearch.dataprepper.plugins.kafka.configuration.SchemaConfig.getBasicAuthCredentialsSource()" because the return value of "org.opensearch.dataprepper.plugins.kafka.source.KafkaSourceConfig.getSchemaConfig()" is null

rendering the source unusable in this configuration.

Environment (please complete the following information):

  • K8s AKS
  • OS Image: CBL-Mariner/Linux
  • Operating System: linux
  • Architecture: amd64
  • Container Runtime Version: containerd://1.6.26

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in data-prepper-plugins/kafka-plugins/src/main/java/org/opensearch/dataprepper/plugins/kafka/util/KafkaSecurityConfigurer.java around lines 207-208, and trace how KafkaSourceConfig handles an absent schema configuration. Reproduce the OAuth-only Kafka source configuration from the issue and verify that the deployment starts without schema registry settings; the relevant Kafka plugin tests are the next place to check for coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka, kubernetes
Domain
authentication, data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.