conductor-oss / conductor-oss/conductor

[Bug] Kafka Event Queue Consumer, Producer, Admin Properties Not Loading Correctly Due to Nested Map Structure

Open
#493 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
32.2k
Forks
1k
Avg merge
2d 5h
Merged PRs (30d)
41

Description

**Describe the bug**
When configuring Kafka consumer, producer, or admin client properties using the conductor.event-queues.kafka prefix, the properties are not being loaded correctly. The properties that contain dots (.) in their names are being converted into nested Map structures instead of being preserved as flat key-value pairs. This prevents the properties from being properly applied to the Kafka clients.

**Details**
Conductor version: v3.21.14
Persistence implementation: MySQL
Queue implementation: MySQL
Lock: Redis
Workflow definition:
```
{
"name": "empty_workflow",
"description": "A minimal workflow that does nothing",
"version": 1,
"tasks": [],
"inputParameters": [],
"outputParameters": {},
"ownerEmail": "owner@email.com",
"schemaVersion": 2
}
```
Task definition: N/A
Event handler definition:
```
{
"name": "sample_kafka_event_consumer",
"event": "kafka:sample-topic",
"condition": "true",
"actions": [
{
"action": "start_workflow",
"start_workflow": {
"name": "sample_workflow",
"version": 1,
"input": {
"payload": "${$}",
"endpoint_url": "http://localhost:8080/api"
}
}
}
],
"active": true
}
```

Code Block: [#KafkaEventQueueProperties.java](https://github.com/conductor-oss/conductor/blob/main/kafka-event-queue/src/main/java/com/netflix/conductor/kafkaeq/config/KafkaEventQueueProperties.java#L165-L169)

**To Reproduce**
1. Steps to reproduce the behavior:
Configure Kafka properties in your application.properties file with properties containing dots, for example:
conductor.event-queues.kafka.consumer.security.protocol=SASL_PLAINTEXT
conductor.event-queues.kafka.consumer.sasl.mechanism=PLAIN
2. Start the Conductor server
3. The properties will be loaded incorrectly as nested maps for consumer in [KafkaEventQueueProperties.java](https://github.com/conductor-oss/conductor/blob/main/kafka-event-queue/src/main/java/com/netflix/conductor/kafkaeq/config/KafkaEventQueueProperties.java:) as below:
```
{
"security": {
"protocol": "SASL_PLAINTEXT"
},
"sasl": {
"mechanism": "PLAIN"
}
}
```

Expected behavior
The properties should be loaded as flat key-value pairs matching the Kafka configuration format as expected in (ConsumerConfig.configNames()):
```
{
"security.protocol": "SASL_PLAINTEXT",
"sasl.mechanism": "PLAIN"
}
```

**Screenshots**
![Image](https://github.com/user-attachments/assets/fb800f1b-a133-4160-9374-18ceed187499)
![Image](https://github.com/user-attachments/assets/37e6ce90-2383-4517-959d-b2f82c34ea87)
![Image](https://github.com/user-attachments/assets/76ec22b7-504f-490e-8aa5-54a6f1f46711)

Additional context:
This issue affects the KafkaEventQueueProperties class where the consumer, producer, and admin properties are defined as Map. The current implementation causes Spring to interpret the dot notation in property names as nested map structures, which prevents the properties from being properly applied to the Kafka clients.

The issue is particularly problematic for Kafka configuration properties that use dot notation, such as:
- security.protocol
- sasl.mechanism
- ssl.truststore.location
- ssl.keystore.location

This prevents proper configuration of security, SSL, and other Kafka client settings that use dot notation in their property names.

The workaround would be to modify the KafkaEventQueueProperties class to handle the nested map structure and flatten it before applying to Kafka clients, or to use a different property binding mechanism that preserves the dot notation in property names.

Contributor guide

Open the contributing guide

Research direction

Start with kafka-event-queue/src/main/java/com/netflix/conductor/kafkaeq/config/KafkaEventQueueProperties.java, especially lines 165-169, and reproduce the binding with consumer properties such as security.protocol and sasl.mechanism. Trace how the consumer, producer, and admin maps are passed to the Kafka clients. Done means dotted Kafka property names remain flat key-value pairs and are applied correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka, spring-boot
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.