confluentinc / confluentinc/parallel-consumer

feature: Option for batch to only contain messages of the same key

Open
#266 10 comments 2 reactions 0 assignees View on GitHub
good first issue
Dominant language
Java
Stars
299
Forks
172
PR merge metrics
No merged PRs in 30d

Description

Currently the batches are just a subset of non batched processing.
I.e.
- key based ordering - all keys in all batches will be unique.
- Unordered - no restrictions

Imagine a scenario where you have a topic where key is the account.

- You have a database per account.
- You want to do batch inserts to the database, where you do a batch of inserts (one insert per message) from within the batch.
- You don't want any messages from a different key, in this batch. All messages in this batch will be in a single external system call.
- Ordered inserting in db is still required. Key ordering is used. No other batch in flight is allowed to contain keys already in an inflight batch.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.