spring-cloud / spring-cloud/spring-cloud-stream
Duplicated SSL config during multiple Kafka binders to different clusters (SSL secured and plaintext-open)
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.1k
- Forks
- 646
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 8
Description
Hey, faced with strange behavior that SSL config from kafka1 binder affects kafka2 binder.
For example, we are working with 2 different kafka clusters:
- Kafka cluster №1 (let's say kafka-secured) that is secured with SSL
- Kafka cluster №2 (let's say kafka-open) that is simply available via plaintext
Application.yml:
spring:
application:
name: local
cloud:
function:
definition: securedConsumer;openConsumer
stream:
bindings:
openConsumer-in-0:
binder: kafka-open
destination: "some-topic-in-kafka-open"
securedConsumer-in-0:
binder: kafka-secured
destination: "some-topic-in-kafka-secured"
binders:
kafka-open:
type: kstream
default-candidate: false
inherit-environment: false
environment:
spring.cloud.stream.kafka.streams:
binder:
brokers: "some-open-kafka-host:9092"
configuration:
security.protocol: PLAINTEXT
bindings:
securedConsumer-in-0:
consumer:
startOffset: latest
application-id: "open-consumer-application-id"
key-serde: 'org.apache.kafka.common.serialization.Serdes$VoidSerde'
value-serde: 'org.apache.kafka.common.serialization.Serdes$StringSerde'
kafka-secured:
type: kstream
default-candidate: false
inherit-environment: false
environment:
spring.cloud.stream.kafka.streams:
binder:
brokers: "some-secured-kafka-host:9092"
configuration:
security.protocol: SSL
ssl:
truststore:
location: "some.truststore.jks"
password: "some-pass"
keystore:
location: "some.keystore.jks"
password: "some-pass"
bindings:
openConsumer-in-0:
consumer:
startOffset: latest
application-id: "secured-consumer-application-id"
key-serde: 'org.apache.kafka.common.serialization.Serdes$VoidSerde'
value-serde: 'org.apache.kafka.common.serialization.Serdes$StringSerde'
Version of the framework
Tested both on spring-cloud-stream-binder-kafka-streams v4.2.0 and v3.2.10
Expected behavior
All streams from/to kafka-open working normally.
Actual behavior
Consumer is not subscribed to partitions and warns about:
org.apache.kafka.clients.NetworkClient : [Consumer clientId=..., groupId=...] Connection to node -3 (...) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.
In o.a.k.clients.consumer.ConsumerConfig : ConsumerConfig values: i see that open-consumer has SSL config from secured-consumer.
Probably, it's merged in org.springframework.cloud.stream.binder.kafka.streams.KafkaStreamsBinderSupportAutoConfiguration#streamConfigGlobalProperties
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at KafkaStreamsBinderSupportAutoConfiguration#streamConfigGlobalProperties and reproduce the two-binder configuration with spring-cloud-stream-binder-kafka-streams v4.2.0 or v3.2.10. Compare the generated ConsumerConfig values for the plaintext and SSL consumers; done means each binder uses only its own security settings and the open consumer subscribes successfully without SSL authentication warnings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka, spring
- Domain
- distributed-systems, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100