apache / apache/rocketmq-externals
[rocketmq-connector-flink] rebalance cause offset rollback to long time ago
- Dominant language
- Java
- Stars
- 4.6k
- Forks
- 3k
- Avg merge
- 2h 33m
- Merged PRs (30d)
- 1
Description
Reproducible Steps:
1.suppose there 3 queue for topic and consumer with two subTasks
2.queue(0,1)consumer by subTask 1 and queue consumer by subTask 2
3. after running for a long while.offset state such as [0:100,1:300,2:120]
4. restart flink job with checkpoint
5. subTask 1 recover with restoredOffsets[0:100,1:300,2:120] like subTask 2.
6. suppose queue(0,1)still consumer by subTask 1 and queue consumer by subTask 2
7. trigger rebalance one time,make subTask 2 consumer queue (0,1,2)
8. running for a while,the offset will be change a lot by subTask 2 ,maye like [0:1000,1:3000,2:1200]
9. subTask 1 offsetTable will be empty after snapshot
10. trigger a rebalance, assign (0, 1) to subTask 1 again
11. In subTask 1,the restored is true and offset table is empty and can't find queue(0, 1),so it will get from restoredOffsets[0:100,1:300,2:120],so will get wrong offset.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in the rocketmq-connector-flink offset restoration and rebalance path, using the numbered checkpoint and queue-assignment reproduction in the issue. Run the restart and rebalance scenario; done means reassigned queues do not fall back to the old restored offsets after the offset table is emptied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100