conductor-oss / conductor-oss/conductor
[Bug] Kafka Event Queue Consumer, Producer, Admin Properties Not Loading Correctly Due to Nested Map Structure
- Dominant language
- Java
- Stars
- 32.2k
- Forks
- 1k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 41
Description
**Describe the bug**
When configuring Kafka consumer, producer, or admin client properties using the conductor.event-queues.kafka prefix, the properties are not being loaded correctly. The properties that contain dots (.) in their names are being converted into nested Map structures instead of being preserved as flat key-value pairs. This prevents the properties from being properly applied to the Kafka clients.
**Details**
Conductor version: v3.21.14
Persistence implementation: MySQL
Queue implementation: MySQL
Lock: Redis
Workflow definition:
```
{
"name": "empty_workflow",
"description": "A minimal workflow that does nothing",
"version": 1,
"tasks": [],
"inputParameters": [],
"outputParameters": {},
"ownerEmail": "owner@email.com",
"schemaVersion": 2
}
```
Task definition: N/A
Event handler definition:
```
{
"name": "sample_kafka_event_consumer",
"event": "kafka:sample-topic",
"condition": "true",
"actions": [
{
"action": "start_workflow",
"start_workflow": {
"name": "sample_workflow",
"version": 1,
"input": {
"payload": "${$}",
"endpoint_url": "http://localhost:8080/api"
}
}
}
],
"active": true
}
```
Code Block: [#KafkaEventQueueProperties.java](https://github.com/conductor-oss/conductor/blob/main/kafka-event-queue/src/main/java/com/netflix/conductor/kafkaeq/config/KafkaEventQueueProperties.java#L165-L169)
**To Reproduce**
1. Steps to reproduce the behavior:
Configure Kafka properties in your application.properties file with properties containing dots, for example:
conductor.event-queues.kafka.consumer.security.protocol=SASL_PLAINTEXT
conductor.event-queues.kafka.consumer.sasl.mechanism=PLAIN
2. Start the Conductor server
3. The properties will be loaded incorrectly as nested maps for consumer in [KafkaEventQueueProperties.java](https://github.com/conductor-oss/conductor/blob/main/kafka-event-queue/src/main/java/com/netflix/conductor/kafkaeq/config/KafkaEventQueueProperties.java:) as below:
```
{
"security": {
"protocol": "SASL_PLAINTEXT"
},
"sasl": {
"mechanism": "PLAIN"
}
}
```
Expected behavior
The properties should be loaded as flat key-value pairs matching the Kafka configuration format as expected in (ConsumerConfig.configNames()):
```
{
"security.protocol": "SASL_PLAINTEXT",
"sasl.mechanism": "PLAIN"
}
```
**Screenshots**



Additional context:
This issue affects the KafkaEventQueueProperties class where the consumer, producer, and admin properties are defined as Map. The current implementation causes Spring to interpret the dot notation in property names as nested map structures, which prevents the properties from being properly applied to the Kafka clients.
The issue is particularly problematic for Kafka configuration properties that use dot notation, such as:
- security.protocol
- sasl.mechanism
- ssl.truststore.location
- ssl.keystore.location
This prevents proper configuration of security, SSL, and other Kafka client settings that use dot notation in their property names.
The workaround would be to modify the KafkaEventQueueProperties class to handle the nested map structure and flatten it before applying to Kafka clients, or to use a different property binding mechanism that preserves the dot notation in property names.
Contributor guide
Research direction
Start with kafka-event-queue/src/main/java/com/netflix/conductor/kafkaeq/config/KafkaEventQueueProperties.java, especially lines 165-169, and reproduce the binding with consumer properties such as security.protocol and sasl.mechanism. Trace how the consumer, producer, and admin maps are passed to the Kafka clients. Done means dotted Kafka property names remain flat key-value pairs and are applied correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka, spring-boot
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100