aio-libs / aio-libs/aiokafka

Refactor Group Coordination

未关闭
#481 0 条评论 2 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
1.4k
派生
269
平均合并
1 天 1 小时
30 天内合并 PR
6

描述

The current implementation is a bunch of hacks made to make sure aiokafka works with the same input parameters as Java's client while remaining asynchronous as much as possible. This was made with the attempt to maintain similarity between Java's internals and aiokafka's. But at this point it changed so drastically I think it's fine to let go that similarity and make sure internals follow asyncio's principals, rather than Java's.

Points I want to be fixed here:
* [] Group coordination should only rejoin the group if we have fetchers, ready to consume that data. This will be similar to what happens with `poll()` in Java's client but will allow clients to just let fetchers hang around during rebalances and still rejoin. (for example, create a task to fetch and only after subscribing to a group)
* [] Remove offset_refresh routines. This was a result of several tries to fix problems while maintaining `subscription` state as the centre of communication between components. At this point let's just let Fetcher do an RPC call on Coordinator. The reference graph would be like:
Consumer -> Fetcher
Consumer -> Coordinator
Consumer -> SubscriptionState
Fetcher -> SubscriptionState
Coordinator -> SubscriptionState
Fetcher -> Coordinator

While still straightforward, but with another link between Fetcher and Coordinator. SubscriptionState should not contain a hard reference on any of the parent objects.
* [] Authorization errors should also communicate in a better way. Probably we should cancel all fetch requests, thus making Coordination stop (no pending fetches). And if another fetch will be created will we rejoin.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。