apache / apache/rocketmq-externals
[rocketmq-connect]sink connector repeat consume message problem
- Dominant language
- Java
- Stars
- 4.6k
- Forks
- 3k
- Avg merge
- 2h 33m
- Merged PRs (30d)
- 1
Description
sink connector单实例与多实例重复消费问题:
1.单实例重复消费
【现象】
- 目前设计是每次runtime启动后消费组名称改变,导致每次启动实例pushConsumer 从offsetTopic上拉取的消息是从0位置开始拉取的,同时拉取到的值会持久化到本地文件,持久化周期是10s,当再次启动后会优先使用本地持久化的offset
- 上述情况就导致了如果某次启动未消费到最新的位置,本地的持久化文件内容就会被改为旧的位置,再次启动就会重复消费
【操作】
- 目前是暂时将消费组设置不变,每次启动后不会出现从0消费现象
【遗留问题】
- 当我在未持久化时异常停机,再次启动会导致有10s的消息重复消费的现象
2.多实例重复消费
【现象】
- 多消费组问题,不同的实例是不同的消费组,导致的问题与上面从0位置消费一致,同时由于是多实例同时监听offsetTopic会导致各自的本地持久化文件内容错乱
- 持久化周期问题,若某个实例在持久化的周期内挂掉,这就导致其他实例负载均衡后再去拉取消息是从上次持久化的时间点位置拉取的消息
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the RocketMQ Connect sink connector runtime, consumer-group setup, and offset persistence handling. Reproduce the documented single-instance restart and multi-instance load-balancing cases, then trace how offsets are read and persisted. Done means restarts and instance changes do not cause messages to be consumed again beyond the intended delivery semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100