apache / apache/rocketmq-externals

[rocketmq-spark] ask a master for help, repeated consumption occurs when the program restarts

Open
#901 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
4.6k
Forks
3k
Avg merge
2h 33m
Merged PRs (30d)
1

Description

Reference code : https://github.com/apache/rocketmq-externals/blob/master/rocketmq-spark/src/test/java/org/apache/rocketmq/spark/streaming/RocketMqUtilsTest.java

![code](https://user-images.githubusercontent.com/38802273/218986039-807f5465-3d45-4f33-8b0a-97b2ea9b526e.png)

My code :

![image](https://user-images.githubusercontent.com/38802273/218986812-fe74ab87-87fd-4040-9116-d8220b3a1303.png)

Consumer Strategy Have chosen lastest, This setting doesn't seem to work, Program restart consumes historical data,

How to solve this problem?

The complete code is as follows:
```
try {

Map optionParams = new HashMap<>();
optionParams.put(RocketMQConfig.NAME_SERVER_ADDR, nameSrvAddr);
SparkConf sparkConf = new SparkConf().setAppName("JavaCustomReceiver").setMaster("local[*]");
JavaStreamingContext sc = new JavaStreamingContext(sparkConf, new Duration(duration));

List topics = new ArrayList<>();
if (StringUtils.hasText(topic)) {
for (String s : topic.split(";")) {
topics.add(s);
}
}

LocationStrategy locationStrategy = LocationStrategy.PreferConsistent();

JavaInputDStream stream = RocketMqUtils.createJavaMQPullStream(sc, groupId,
topics, ConsumerStrategy.lastest(), false, false, false, locationStrategy, optionParams);

stream.foreachRDD(new VoidFunction>() {

private static final long serialVersionUID = 1L;

@Override
public void call(JavaRDD messageExtJavaRDD) throws Exception {

JavaRDD GPSRDDJavaRDD = messageExtJavaRDD.map(new Function() {

private static final long serialVersionUID = 1L;

@Override
public GPSRDD call(MessageExt messageExt) throws Exception {

GPSRDD gps = new GPSRDD();
String xxx = new String(messageExt.getBody());
System.out.println(xxx);
return gps;
}
});

}
});

sc.start();

} catch (Exception e) {
e.printStackTrace();
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with rocketmq-spark/src/test/java/org/apache/rocketmq/spark/streaming/RocketMqUtilsTest.java and trace RocketMqUtils.createJavaMQPullStream with ConsumerStrategy.lastest(). Reproduce the restart scenario from the provided code, then determine the expected behavior and update the relevant test or implementation once the cause of repeated historical consumption is established.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
distributed-systems, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.