aio-libs / aio-libs/aiokafka

Document how to handle long message processing with max_poll_interval_ms

Đang mở
#848 3 bình luận 0 reaction 0 người được giao Xem trên GitHub
enhancement
Ngôn ngữ chính
Python
Star
1.4k
Fork
269
Merge trung bình
1 ngày 1 giờ
Pull request đã merge (30 ngày)
6

Mô tả

**Describe the solution you'd like**
I think it'd be good to give a clear warning in the docs about processing long messages and focus people's attention on max_poll_interval_ms. Maybe another section - "long processing" - in [here](https://aiokafka.readthedocs.io/en/stable/consumer.html#consumer-usage) would be good.

**The problem that I had**
- A message was taking a long time to process.
- After 5 minutes, the consumer doing the processing left the group.
- The group rebalanced
- Another consumer got the same message to process.
- The original consumer finished processing (after 7 minutes), tried to commit the offset, failed with UnknownMemberID, which also causes a rebalance of the whole group, preventing the second consumer from commiting the message.
- That looped forever.

**My understanding of max_poll_interval_ms**
The number isn't sent to Kafka.
It's only used internally in aiokafka to leave the consumer group if the consumer didn't poll for messages in a while, suggesting it's stuck.

Is that correct?

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.