apache / apache/rocketmq-externals

[rocketmq-connector-flink] rebalance cause offset rollback to long time ago

Open
#671 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
4.6k
Forks
3k
Avg merge
2h 33m
Merged PRs (30d)
1

Description

Reproducible Steps:
1.suppose there 3 queue for topic and consumer with two subTasks
2.queue(0,1)consumer by subTask 1 and queue consumer by subTask 2
3. after running for a long while.offset state such as [0:100,1:300,2:120]
4. restart flink job with checkpoint
5. subTask 1 recover with restoredOffsets[0:100,1:300,2:120] like subTask 2.
6. suppose queue(0,1)still consumer by subTask 1 and queue consumer by subTask 2
7. trigger rebalance one time,make subTask 2 consumer queue (0,1,2)
8. running for a while,the offset will be change a lot by subTask 2 ,maye like [0:1000,1:3000,2:1200]
9. subTask 1 offsetTable will be empty after snapshot
10. trigger a rebalance, assign (0, 1) to subTask 1 again
11. In subTask 1,the restored is true and offset table is empty and can't find queue(0, 1),so it will get from restoredOffsets[0:100,1:300,2:120],so will get wrong offset.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in the rocketmq-connector-flink offset restoration and rebalance path, using the numbered checkpoint and queue-assignment reproduction in the issue. Run the restart and rebalance scenario; done means reassigned queues do not fall back to the old restored offsets after the offset table is emptied.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.