AdminClient / KafkaAdmin with merged standard kafka and binder properties

@skomarica commented on Mon Dec 13 2021

Hi,

First of all, my apologies if this was already answered elsewhere; I reviewed the docs, SO Q&A and previous similar issues, but couldn't find the answer.

We use Kafka binder for our spring boot / cloud based services. The complete configuration is based on spring.cloud.stream.* properties and everything has been working as expected so far.

Now, I want to use the AdminClient to verify certain things on the cluster (e.g. list/describe topics). I tried using an autowired KafkaAdmin bean to provide the configuration properties, like:

        try (final AdminClient adminClient = AdminClient.create(kafkaAdmin.getConfigurationProperties())) {

, but these configuration properties don't seem to pick up any of the binder properties (e.g. spring.cloud.stream.kafka.binder.brokers, spring.cloud.stream.kafka.binder.configuration.*...). I also looked at the KafkaAdmin bean implementation and configuration, and it doesn't look like the class is supposed to do any kind of properties merging. If this is the case, it makes the auto-configured KafkaAdmin bean a bit pointless.

I see that the internal AdminClient used by KafkaTopicProvisioner gets initialised correctly, but there is a custom internal logic in the constructor of that class, normalising and merging the standard kafka properties with the kafka binder ones. Most of these classes and beans are internal / package private, and we can't easily use them.

The question is - is there a suggested way to create an AdminClient that will be correctly initialised, respecting all the spring.cloud.stream.kafka.binder.* properties?

@garyrussell commented on Mon Dec 13 2021

This is currently not supported; the auto-configured (by Boot) KafkaAdmin only uses the Boot properties.

Perhaps the KafkaTopicProvisioner could be modified to provide an API to retrieve the merged admin properties for each binding.

Start by reading the KafkaTopicProvisioner initialization logic and the auto-configured KafkaAdmin behavior described in the issue. Trace how binder broker and configuration properties are merged, then define an exposed API or documented approach whose completion can be verified by creating an AdminClient that honors those binder properties.