apache / apache/rocketmq-externals

[RocketMQ-Flink] Job operator need more information not only key and body

Open
#356 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
4.6k
Forks
3k
Avg merge
2h 33m
Merged PRs (30d)
1

Description

**FEATURE REQUEST**

1. Please describe the feature you are requesting.

In a classical scene, a stream pipeline may look like

```
[Producer SendMessage] -> [Flink Job] -> [Downstream Operation] -> [Final Result Data/Event]
```

Every step consumes time then contribute to the latency. To statics/monitor the end to end time consumption, we want to keep the `BornTime` of messages as `BeginTime` of the whole pipeline.

2. Provide any additional detail on your proposed use case for this feature.

In the `RocketMQSource.class`, information about messages are discarded, but only `keys` and `body` of messages saved.
```
for (MessageExt msg : messages) {
byte[] key = msg.getKeys() != null ? msg.getKeys().getBytes(StandardCharsets.UTF_8) : null;
byte[] value = msg.getBody();
OUT data = schema.deserializeKeyAndValue(key, value);

// output and state update are atomic
synchronized (lock) {
context.collectWithTimestamp(data, msg.getBornTimestamp());
}
}
```

I wonder if let users customize the serialization method will be better?

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

should-have

4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:
No

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading RocketMQSource.class and the MessageExt handling around schema.deserializeKeyAndValue. Trace how keys, body, and msg.getBornTimestamp() reach the Flink job, then clarify the serialization contract needed to preserve additional message information. Done means the proposed customization supports the end-to-end timing use case without losing the existing key and body behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.