Destination Kafka : Enable partition key configuration to preserve Kafka ordering
- Dominant language
- Python
- Stars
- 22.1k
- Forks
- 5.3k
- PR merge metrics
- PR metrics pending
Description
### Connector Name
destination-kafka
### Connector Version
0.1.11
### What step the error happened?
None
### Relevant information
## Problem
The Kafka destination connector currently assigns a random UUID as the message key for each record.
This results in:
Records being distributed across partitions arbitrarily
Related events not being routed to the same partition
Loss of ordering guarantees for entity-level data
Since Kafka guarantees ordering only within a partition, this makes it unsuitable for use cases that rely on ordered event processing.
## Example
```
### Input Events
{ "user_id": "U1", "order_id": "O1", "status": "CREATED" }
{ "user_id": "U1", "order_id": "O1", "status": "SHIPPED" }
{ "user_id": "U1", "order_id": "O1", "status": "DELIVERED" }
### Current Behavior (random UUID key)
Each record gets a different key
Events may go to different partitions
While consuming from kafka topic
#### Possible observed sequences:
SHIPPED → CREATED → DELIVERED
DELIVERED → SHIPPED → CREATED
CREATED → SHIPPED → DELIVERED
CREATED → DELIVERED → SHIPPED
#### Expected sequence:
CREATED → SHIPPED → DELIVERED
```
## Proposed Solution
Add a new optional configuration field:
"partition_key_field": "user_id"
### Behavior
Use the specified field(s) to construct the Kafka message key
If partition_key_field is not specified, fallback to the current random uuid message key approach
For composite keys, serialize or hash the values into a single key
### Supported cases
Single field: "user_id"
Nested field: "user.id"
Composite keys: "user_id,order_id"
### Relevant log output
```shell
```
### Contribute
- [x] Yes, I want to contribute
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11740
Contributor guide
Assessment
This issue has not been assessed yet.