aio-libs / aio-libs/aiokafka

LeaveGroup but failed to rejoin

未关闭
#625 14 条评论 7 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
1.4k
派生
269
平均合并
1 天 1 小时
30 天内合并 PR
6

描述

**Describe the bug**
Randomly, kafka consumer failed to fetch new messages, then kicked out of the consumer group after max_poll timeout, but failed to rejoin

**Expected behaviour**
The consumer should not fail to fetch new messages

**Environment (please complete the following information):**
- aiokafka version 0.6.0
- Kafka 2.2.1 and 2.3.1, same error happened in both versions

**Reproducible example**
Difficult to reproduce, it happens occasionally on our live environment

What we observed
- Out of sudden, one consumer stops fetching messages for its assigned partition. The lags increase. We observe the consumer is still registered in the Kafka broker properly
- After the max_poll timeout, the broker forces rebalance and kick the consumer out
- The consumer still thinks it has the assigned partition, although it receives LeaveGroup and revoked partition. It keeps sending packages to the Broker (we observe a very high RX/TX caused by this consumer)
- We can only detect the abnormality and manually stop the consumer.

** logs **

* after max_poll timeout, we saw the following messages
INFO] aiokafka.consumer.group_coordinator => LeaveGroup request succeeded
12:11:45
2020-06-06 12:11:45,510 [ERROR] aiokafka.consumer.group_coordinator => OffsetCommit failed for group score_worker due to group error ([Error 25] UnknownMemberIdError: score_worker), will rejoin
12:11:45
2020-06-06 12:11:45,517 [ERROR] aiokafka.consumer.group_coordinator => OffsetCommit failed for group score_worker due to group error ([Error 25] UnknownMemberIdError: score_worker), will rejoin
2020-06-06 12:11:45,517 [ERROR] aiokafka.consumer.group_coordinator => OffsetCommit failed for group score_worker due to group error ([Error 25] UnknownMemberIdError: score_worker), will rejoin
12:11:45
2020-06-06 12:11:45,523 [ERROR] aiokafka.consumer.group_coordinator => OffsetCommit failed for group score_worker due to group error ([Error 25] UnknownMemberIdError: score_worker), will rejoin
2020-06-06 12:11:45,523 [ERROR] aiokafka.consumer.group_coordinator => OffsetCommit failed for group score_worker due to group error ([Error 25] UnknownMemberIdError: score_worker), will rejoin
* ... repeats the UnknownMemberId for a number of times*

12:11:45
2020-06-06 12:11:45,551 [ERROR] aiokafka.consumer.group_coordinator => OffsetCommit failed before join, ignoring: CommitFailedError: ('Commit cannot be completed since the group has already\n rebalanced and assigned the partitions to another member.\n This means that the time between subsequent calls to poll()\n was longer than the configured max_poll_interval_ms,

12:11:45
2020-06-06 12:11:45,556 [INFO] aiokafka.consumer.group_coordinator => Revoking previously assigned partitions frozenset({TopicPartition(topic='pron-job', partition=39), TopicPartition(topic='pron-job', partition=75), TopicPartition(topic='pron-job', partition=111), TopicPartition(topic='pron-job', partition=3)}) for group score_worker
2020-06-06 12:11:45,556 [INFO] aiokafka.consumer.group_coordinator => Revoking previously assigned partitions frozenset({TopicPartition(topic='pron-job', partition=39), TopicPartition(topic='pron-job', partition=75), TopicPartition(topic='pron-job', partition=111), TopicPartition(topic='pron-job', partition=3)}) for group score_worker

After that, there was no INFO level log at all. But the length of consumer.assignment is still the number before revoked.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。