Destination Kafka : Enable partition key configuration to preserve Kafka ordering
- Linguagem predominante
- Python
- Estrelas
- 22.1k
- Forks
- 5.4k
- Merge médio
- 5h
- PRs com merge (30d)
- 671
Descrição
### Connector Name
destination-kafka
### Connector Version
0.1.11
### What step the error happened?
None
### Relevant information
## Problem
The Kafka destination connector currently assigns a random UUID as the message key for each record.
This results in:
Records being distributed across partitions arbitrarily
Related events not being routed to the same partition
Loss of ordering guarantees for entity-level data
Since Kafka guarantees ordering only within a partition, this makes it unsuitable for use cases that rely on ordered event processing.
## Example
```
### Input Events
{ "user_id": "U1", "order_id": "O1", "status": "CREATED" }
{ "user_id": "U1", "order_id": "O1", "status": "SHIPPED" }
{ "user_id": "U1", "order_id": "O1", "status": "DELIVERED" }
### Current Behavior (random UUID key)
Each record gets a different key
Events may go to different partitions
While consuming from kafka topic
#### Possible observed sequences:
SHIPPED → CREATED → DELIVERED
DELIVERED → SHIPPED → CREATED
CREATED → SHIPPED → DELIVERED
CREATED → DELIVERED → SHIPPED
#### Expected sequence:
CREATED → SHIPPED → DELIVERED
```
## Proposed Solution
Add a new optional configuration field:
"partition_key_field": "user_id"
### Behavior
Use the specified field(s) to construct the Kafka message key
If partition_key_field is not specified, fallback to the current random uuid message key approach
For composite keys, serialize or hash the values into a single key
### Supported cases
Single field: "user_id"
Nested field: "user.id"
Composite keys: "user_id,order_id"
### Relevant log output
```shell
```
### Contribute
- [x] Yes, I want to contribute
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11740
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.