Refactor Group Coordination
- Lingua principale
- Python
- Stelle
- 1.4k
- Fork
- 269
- Merge medio
- 1g 1h
- PR unite (30g)
- 6
Descrizione
The current implementation is a bunch of hacks made to make sure aiokafka works with the same input parameters as Java's client while remaining asynchronous as much as possible. This was made with the attempt to maintain similarity between Java's internals and aiokafka's. But at this point it changed so drastically I think it's fine to let go that similarity and make sure internals follow asyncio's principals, rather than Java's.
Points I want to be fixed here:
* [] Group coordination should only rejoin the group if we have fetchers, ready to consume that data. This will be similar to what happens with `poll()` in Java's client but will allow clients to just let fetchers hang around during rebalances and still rejoin. (for example, create a task to fetch and only after subscribing to a group)
* [] Remove offset_refresh routines. This was a result of several tries to fix problems while maintaining `subscription` state as the centre of communication between components. At this point let's just let Fetcher do an RPC call on Coordinator. The reference graph would be like:
Consumer -> Fetcher
Consumer -> Coordinator
Consumer -> SubscriptionState
Fetcher -> SubscriptionState
Coordinator -> SubscriptionState
Fetcher -> Coordinator
While still straightforward, but with another link between Fetcher and Coordinator. SubscriptionState should not contain a hard reference on any of the parent objects.
* [] Authorization errors should also communicate in a better way. Probably we should cancel all fetch requests, thus making Coordination stop (no pending fetches). And if another fetch will be created will we rejoin.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.