aio-libs / aio-libs/aiokafka

Refactor Group Coordination

Ouverte
#481 0 commentaires 2 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Python
Étoiles
1.4k
Forks
269
Merge moyen
1 j 1 h
PR mergées (30 j)
6

Description

The current implementation is a bunch of hacks made to make sure aiokafka works with the same input parameters as Java's client while remaining asynchronous as much as possible. This was made with the attempt to maintain similarity between Java's internals and aiokafka's. But at this point it changed so drastically I think it's fine to let go that similarity and make sure internals follow asyncio's principals, rather than Java's.

Points I want to be fixed here:
* [] Group coordination should only rejoin the group if we have fetchers, ready to consume that data. This will be similar to what happens with `poll()` in Java's client but will allow clients to just let fetchers hang around during rebalances and still rejoin. (for example, create a task to fetch and only after subscribing to a group)
* [] Remove offset_refresh routines. This was a result of several tries to fix problems while maintaining `subscription` state as the centre of communication between components. At this point let's just let Fetcher do an RPC call on Coordinator. The reference graph would be like:
Consumer -> Fetcher
Consumer -> Coordinator
Consumer -> SubscriptionState
Fetcher -> SubscriptionState
Coordinator -> SubscriptionState
Fetcher -> Coordinator

While still straightforward, but with another link between Fetcher and Coordinator. SubscriptionState should not contain a hard reference on any of the parent objects.
* [] Authorization errors should also communicate in a better way. Probably we should cancel all fetch requests, thus making Coordination stop (no pending fetches). And if another fetch will be created will we rejoin.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.