spring-cloud / spring-cloud/spring-cloud-stream

Duplicated SSL config during multiple Kafka binders to different clusters (SSL secured and plaintext-open)

Open
#3,068 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
1.1k
Forks
646
Avg merge
2d 3h
Merged PRs (30d)
8

Description

Hey, faced with strange behavior that SSL config from kafka1 binder affects kafka2 binder.
For example, we are working with 2 different kafka clusters:

  • Kafka cluster №1 (let's say kafka-secured) that is secured with SSL
  • Kafka cluster №2 (let's say kafka-open) that is simply available via plaintext

Application.yml:

spring:
  application:
    name: local
  cloud:
    function:
      definition: securedConsumer;openConsumer
    stream:
      bindings:
        openConsumer-in-0:
          binder: kafka-open
          destination: "some-topic-in-kafka-open"
        securedConsumer-in-0:
          binder: kafka-secured
          destination: "some-topic-in-kafka-secured"
      binders:
        kafka-open:
          type: kstream
          default-candidate: false
          inherit-environment: false
          environment:
            spring.cloud.stream.kafka.streams:
              binder:
                brokers: "some-open-kafka-host:9092"
                configuration:
                  security.protocol: PLAINTEXT
              bindings:
                securedConsumer-in-0:
                  consumer:
                    startOffset: latest
                    application-id: "open-consumer-application-id"
                    key-serde: 'org.apache.kafka.common.serialization.Serdes$VoidSerde'
                    value-serde: 'org.apache.kafka.common.serialization.Serdes$StringSerde'
        kafka-secured:
          type: kstream
          default-candidate: false
          inherit-environment: false
          environment:
            spring.cloud.stream.kafka.streams:
              binder:
                brokers: "some-secured-kafka-host:9092"
                configuration:
                  security.protocol: SSL
                  ssl:
                    truststore:
                      location: "some.truststore.jks"
                      password: "some-pass"
                    keystore:
                      location: "some.keystore.jks"
                      password: "some-pass"
              bindings:
                openConsumer-in-0:
                  consumer:
                    startOffset: latest
                    application-id: "secured-consumer-application-id"
                    key-serde: 'org.apache.kafka.common.serialization.Serdes$VoidSerde'
                    value-serde: 'org.apache.kafka.common.serialization.Serdes$StringSerde'

Version of the framework
Tested both on spring-cloud-stream-binder-kafka-streams v4.2.0 and v3.2.10

Expected behavior
All streams from/to kafka-open working normally.

Actual behavior
Consumer is not subscribed to partitions and warns about:
org.apache.kafka.clients.NetworkClient : [Consumer clientId=..., groupId=...] Connection to node -3 (...) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.

In o.a.k.clients.consumer.ConsumerConfig : ConsumerConfig values: i see that open-consumer has SSL config from secured-consumer.
Probably, it's merged in org.springframework.cloud.stream.binder.kafka.streams.KafkaStreamsBinderSupportAutoConfiguration#streamConfigGlobalProperties

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at KafkaStreamsBinderSupportAutoConfiguration#streamConfigGlobalProperties and reproduce the two-binder configuration with spring-cloud-stream-binder-kafka-streams v4.2.0 or v3.2.10. Compare the generated ConsumerConfig values for the plaintext and SSL consumers; done means each binder uses only its own security settings and the open consumer subscribes successfully without SSL authentication warnings.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka, spring
Domain
distributed-systems, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.