[Question][scalable-topics] How are consumer assignments ordered across controller reconnects?
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
## Context
PIP-468 allows consumer membership changes to recompute assignments without changing the segment layout. Different `ScalableConsumerAssignment` messages can therefore carry the same `layout_epoch`.
The current protocol carries the layout epoch and segment list, but no assignment generation, controller term, or session incarnation. The Java V5 client rejects only `epoch < currentEpoch`, so changed equal-epoch assignments are deliberately accepted.
Within one live `ClientCnx`, wire order gives clients an ordering. Across controller reconnect or leader failover, I could not find a protocol-level fencing rule.
There is another ordering detail in the current broker path: `registerConsumer` can push an assignment update before `ServerCnx` writes the subscribe response, so the response does not appear to be an obvious synchronization barrier.
## Questions
1. Within one controller connection, should clients apply changed equal-epoch assignments strictly in receive order?
2. Is `CommandScalableTopicSubscribeResponse` a baseline that precedes all pushed updates, or may a push received before the response represent newer state?
3. After reconnect or controller leader failover, can a callback or frame from the previous connection race the new connection's initial assignment?
4. Should clients locally fence all callbacks by connection incarnation?
5. Is an assignment generation or controller/session term planned for the wire protocol?
Concrete example: assignments A and B both have `layout_epoch = 7`, but differ because a consumer joined. If A comes from the old connection and B from the new connection, is there any protocol-supported way to determine which is authoritative?
References: PIP-468, `SubscriptionCoordinator.computeAssignment`, `ScalableConsumerClient.onAssignmentUpdate`, and `ServerCnx.handleCommandScalableTopicSubscribe` on `v5.0.0-M1` and current `master`.
Contributor guide
Research direction
Start with PIP-468 and trace SubscriptionCoordinator.computeAssignment, ScalableConsumerClient.onAssignmentUpdate, and ServerCnx.handleCommandScalableTopicSubscribe on v5.0.0-M1 and current master. Compare subscribe-response ordering, pushed updates, and reconnect behavior across the referenced paths. Done means documenting the authoritative ordering and fencing rule, or identifying the required protocol change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100