airbytehq / airbytehq/airbyte

Destination Kafka : Enable partition key configuration to preserve Kafka ordering

Open
#75,292 5 comments 0 reactions 0 assignees View on GitHub
area/connectors autoteam community connectors/destination/kafka needs-triage team/extensibility type/bug
Dominant language
Python
Stars
22.1k
Forks
5.3k
PR merge metrics
PR metrics pending

Description

### Connector Name

destination-kafka

### Connector Version

0.1.11

### What step the error happened?

None

### Relevant information

## Problem

The Kafka destination connector currently assigns a random UUID as the message key for each record.

This results in:

Records being distributed across partitions arbitrarily
Related events not being routed to the same partition
Loss of ordering guarantees for entity-level data

Since Kafka guarantees ordering only within a partition, this makes it unsuitable for use cases that rely on ordered event processing.

## Example

```
### Input Events
{ "user_id": "U1", "order_id": "O1", "status": "CREATED" }
{ "user_id": "U1", "order_id": "O1", "status": "SHIPPED" }
{ "user_id": "U1", "order_id": "O1", "status": "DELIVERED" }

### Current Behavior (random UUID key)
Each record gets a different key
Events may go to different partitions

While consuming from kafka topic
#### Possible observed sequences:
SHIPPED → CREATED → DELIVERED
DELIVERED → SHIPPED → CREATED
CREATED → SHIPPED → DELIVERED
CREATED → DELIVERED → SHIPPED

#### Expected sequence:
CREATED → SHIPPED → DELIVERED
```

## Proposed Solution

Add a new optional configuration field:
"partition_key_field": "user_id"

### Behavior
Use the specified field(s) to construct the Kafka message key
If partition_key_field is not specified, fallback to the current random uuid message key approach
For composite keys, serialize or hash the values into a single key

### Supported cases
Single field: "user_id"
Nested field: "user.id"
Composite keys: "user_id,order_id"

### Relevant log output

```shell

```

### Contribute

- [x] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11740

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.