A max_records_per_partition parameter for Consumer.getmany
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 269
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 6
Description
`Consumer.getmany` already has a `max_records` parameter to to limit the number of messages returned, but nothing to limit the number of records by partition. I've implemented a `max_records_per_partition` to provide such limit: https://github.com/aio-libs/aiokafka/compare/master...aure-olli:max_records_per_partition
My use case is for my topic merger: I have to keep one message per partition in order to compare them, and multiple calls to `Consumer.getone` means many useless `if partitions and tp not in partitions:` test (https://github.com/aure-olli/aiokafka/blob/master/test_merge.py#L312:L322).
The usage of `max_records_per_partition` as a `dict` (with one limit for each partition) could probably cover more cases.
Contributor guide
Assessment
This issue has not been assessed yet.