apache / apache/rocketmq-externals
[rocketmq-spark] ask a master for help, repeated consumption occurs when the program restarts
- Dominant language
- Java
- Stars
- 4.6k
- Forks
- 3k
- Avg merge
- 2h 33m
- Merged PRs (30d)
- 1
Description
Reference code : https://github.com/apache/rocketmq-externals/blob/master/rocketmq-spark/src/test/java/org/apache/rocketmq/spark/streaming/RocketMqUtilsTest.java

My code :

Consumer Strategy Have chosen lastest, This setting doesn't seem to work, Program restart consumes historical data,
How to solve this problem?
The complete code is as follows:
```
try {
Map optionParams = new HashMap<>();
optionParams.put(RocketMQConfig.NAME_SERVER_ADDR, nameSrvAddr);
SparkConf sparkConf = new SparkConf().setAppName("JavaCustomReceiver").setMaster("local[*]");
JavaStreamingContext sc = new JavaStreamingContext(sparkConf, new Duration(duration));
List topics = new ArrayList<>();
if (StringUtils.hasText(topic)) {
for (String s : topic.split(";")) {
topics.add(s);
}
}
LocationStrategy locationStrategy = LocationStrategy.PreferConsistent();
JavaInputDStream stream = RocketMqUtils.createJavaMQPullStream(sc, groupId,
topics, ConsumerStrategy.lastest(), false, false, false, locationStrategy, optionParams);
stream.foreachRDD(new VoidFunction>() {
private static final long serialVersionUID = 1L;
@Override
public void call(JavaRDD messageExtJavaRDD) throws Exception {
JavaRDD GPSRDDJavaRDD = messageExtJavaRDD.map(new Function() {
private static final long serialVersionUID = 1L;
@Override
public GPSRDD call(MessageExt messageExt) throws Exception {
GPSRDD gps = new GPSRDD();
String xxx = new String(messageExt.getBody());
System.out.println(xxx);
return gps;
}
});
}
});
sc.start();
} catch (Exception e) {
e.printStackTrace();
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with rocketmq-spark/src/test/java/org/apache/rocketmq/spark/streaming/RocketMqUtilsTest.java and trace RocketMqUtils.createJavaMQPullStream with ConsumerStrategy.lastest(). Reproduce the restart scenario from the provided code, then determine the expected behavior and update the relevant test or implementation once the cause of repeated historical consumption is established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- distributed-systems, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100