apache / apache/rocketmq-externals
[RocketMQ-Flink] Job operator need more information not only key and body
- Dominant language
- Java
- Stars
- 4.6k
- Forks
- 3k
- Avg merge
- 2h 33m
- Merged PRs (30d)
- 1
Description
**FEATURE REQUEST**
1. Please describe the feature you are requesting.
In a classical scene, a stream pipeline may look like
```
[Producer SendMessage] -> [Flink Job] -> [Downstream Operation] -> [Final Result Data/Event]
```
Every step consumes time then contribute to the latency. To statics/monitor the end to end time consumption, we want to keep the `BornTime` of messages as `BeginTime` of the whole pipeline.
2. Provide any additional detail on your proposed use case for this feature.
In the `RocketMQSource.class`, information about messages are discarded, but only `keys` and `body` of messages saved.
```
for (MessageExt msg : messages) {
byte[] key = msg.getKeys() != null ? msg.getKeys().getBytes(StandardCharsets.UTF_8) : null;
byte[] value = msg.getBody();
OUT data = schema.deserializeKeyAndValue(key, value);
// output and state update are atomic
synchronized (lock) {
context.collectWithTimestamp(data, msg.getBornTimestamp());
}
}
```
I wonder if let users customize the serialization method will be better?
2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
should-have
4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:
No
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading RocketMQSource.class and the MessageExt handling around schema.deserializeKeyAndValue. Trace how keys, body, and msg.getBornTimestamp() reach the Flink job, then clarify the serialization contract needed to preserve additional message information. Done means the proposed customization supports the end-to-end timing use case without losing the existing key and body behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100