aio-libs / aio-libs/aiokafka

A max_records_per_partition parameter for Consumer.getmany

Open
#525 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.4k
Forks
269
Avg merge
1d 1h
Merged PRs (30d)
6

Description

`Consumer.getmany` already has a `max_records` parameter to to limit the number of messages returned, but nothing to limit the number of records by partition. I've implemented a `max_records_per_partition` to provide such limit: https://github.com/aio-libs/aiokafka/compare/master...aure-olli:max_records_per_partition

My use case is for my topic merger: I have to keep one message per partition in order to compare them, and multiple calls to `Consumer.getone` means many useless `if partitions and tp not in partitions:` test (https://github.com/aure-olli/aiokafka/blob/master/test_merge.py#L312:L322).

The usage of `max_records_per_partition` as a `dict` (with one limit for each partition) could probably cover more cases.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.