LeaveGroup but failed to rejoin
- 主要语言
- Python
- 星标
- 1.4k
- 派生
- 269
- 平均合并
- 1 天 1 小时
- 30 天内合并 PR
- 6
描述
**Describe the bug**
Randomly, kafka consumer failed to fetch new messages, then kicked out of the consumer group after max_poll timeout, but failed to rejoin
**Expected behaviour**
The consumer should not fail to fetch new messages
**Environment (please complete the following information):**
- aiokafka version 0.6.0
- Kafka 2.2.1 and 2.3.1, same error happened in both versions
**Reproducible example**
Difficult to reproduce, it happens occasionally on our live environment
What we observed
- Out of sudden, one consumer stops fetching messages for its assigned partition. The lags increase. We observe the consumer is still registered in the Kafka broker properly
- After the max_poll timeout, the broker forces rebalance and kick the consumer out
- The consumer still thinks it has the assigned partition, although it receives LeaveGroup and revoked partition. It keeps sending packages to the Broker (we observe a very high RX/TX caused by this consumer)
- We can only detect the abnormality and manually stop the consumer.
** logs **
* after max_poll timeout, we saw the following messages
INFO] aiokafka.consumer.group_coordinator => LeaveGroup request succeeded
12:11:45
2020-06-06 12:11:45,510 [ERROR] aiokafka.consumer.group_coordinator => OffsetCommit failed for group score_worker due to group error ([Error 25] UnknownMemberIdError: score_worker), will rejoin
12:11:45
2020-06-06 12:11:45,517 [ERROR] aiokafka.consumer.group_coordinator => OffsetCommit failed for group score_worker due to group error ([Error 25] UnknownMemberIdError: score_worker), will rejoin
2020-06-06 12:11:45,517 [ERROR] aiokafka.consumer.group_coordinator => OffsetCommit failed for group score_worker due to group error ([Error 25] UnknownMemberIdError: score_worker), will rejoin
12:11:45
2020-06-06 12:11:45,523 [ERROR] aiokafka.consumer.group_coordinator => OffsetCommit failed for group score_worker due to group error ([Error 25] UnknownMemberIdError: score_worker), will rejoin
2020-06-06 12:11:45,523 [ERROR] aiokafka.consumer.group_coordinator => OffsetCommit failed for group score_worker due to group error ([Error 25] UnknownMemberIdError: score_worker), will rejoin
* ... repeats the UnknownMemberId for a number of times*
12:11:45
2020-06-06 12:11:45,551 [ERROR] aiokafka.consumer.group_coordinator => OffsetCommit failed before join, ignoring: CommitFailedError: ('Commit cannot be completed since the group has already\n rebalanced and assigned the partitions to another member.\n This means that the time between subsequent calls to poll()\n was longer than the configured max_poll_interval_ms,
12:11:45
2020-06-06 12:11:45,556 [INFO] aiokafka.consumer.group_coordinator => Revoking previously assigned partitions frozenset({TopicPartition(topic='pron-job', partition=39), TopicPartition(topic='pron-job', partition=75), TopicPartition(topic='pron-job', partition=111), TopicPartition(topic='pron-job', partition=3)}) for group score_worker
2020-06-06 12:11:45,556 [INFO] aiokafka.consumer.group_coordinator => Revoking previously assigned partitions frozenset({TopicPartition(topic='pron-job', partition=39), TopicPartition(topic='pron-job', partition=75), TopicPartition(topic='pron-job', partition=111), TopicPartition(topic='pron-job', partition=3)}) for group score_worker
After that, there was no INFO level log at all. But the length of consumer.assignment is still the number before revoked.
贡献指南
评估
这个 Issue 还没有评估数据。