apache / apache/doris

[Feature] Kafka Share Group support for Routine Load

Open
#63,190 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

Kafka 4.0 introduced Share Groups (KIP-932): a consumption model where multiple consumers in a group share access to the same partitions without exclusive ownership. The broker tracks per-record delivery state (ACQUIRED → ACCEPTED/RELEASED/REJECTED) with an acquisition lock timeout as the crash-safety mechanism.

Doris Routine Load supports **traditional** Kafka's assign() API where **Parallelism is capped by partition count and Partition skew is unaddressable.**

If we have support for Kafka Queue Semantics (Share group implementation) then it will benefits the usecases of: append-only tables, log/metrics ingestion, high-throughput order-insensitive workloads.

## Proposal

Add an optional LOAD_MODE = "share_group" property to CREATE ROUTINE LOAD. When set:

• divideRoutineLoadJob() creates N tasks with no partition assignment — all tasks subscribe to the same topic via KafkaShareConsumer with the same group.id (job id).
• The BE task loop becomes a single sequential poll() → pipe → commit_txn → ack cycle per task thread, replacing the current N-thread BlockingQueue model.
• On Doris txn commit success: send ACCEPT ack + commitSync().
• On Doris txn commit failure: send RELEASE ack + commitSync() — broker requeues records, next poll gets them back.

Discussion : https://github.com/apache/doris/discussions/63265

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.