apache / apache/rocketmq-externals

Is needed to convert MessageExt body to String in WorkerSinkTask convertToSinkDataEntry method.

Open
#799 0 comments 0 reactions 0 assignees View on GitHub
discuss
Dominant language
Java
Stars
4.6k
Forks
3k
Avg merge
2h 33m
Merged PRs (30d)
1

Description

```java

private SinkDataEntry convertToSinkDataEntry(MessageExt message) {
Map properties = message.getProperties();
String queueName;
EntryType entryType;
Schema schema;
Long timestamp;
Object[] datas = new Object[1];
if (null == recordConverter || recordConverter instanceof RocketMQConverter) {
queueName = properties.get(RuntimeConfigDefine.CONNECT_TOPICNAME);
String connectEntryType = properties.get(RuntimeConfigDefine.CONNECT_ENTRYTYPE);
entryType = StringUtils.isNotEmpty(connectEntryType) ? EntryType.valueOf(connectEntryType) : null;
String connectTimestamp = properties.get(RuntimeConfigDefine.CONNECT_TIMESTAMP);
timestamp = StringUtils.isNotEmpty(connectTimestamp) ? Long.valueOf(connectTimestamp) : null;
String connectSchema = properties.get(RuntimeConfigDefine.CONNECT_SCHEMA);
schema = StringUtils.isNotEmpty(connectSchema) ? JSON.parseObject(connectSchema, Schema.class) : null;
datas = new Object[1];
datas[0] = new String(message.getBody());
} else {
final byte[] messageBody = message.getBody();
final SourceDataEntry sourceDataEntry = JSON.parseObject(new String(messageBody), SourceDataEntry.class);
final Object[] payload = sourceDataEntry.getPayload();
final byte[] decodeBytes = Base64.getDecoder().decode((String) payload[0]);
Object recodeObject;
if (recordConverter instanceof JsonConverter) {
JsonConverter jsonConverter = (JsonConverter) recordConverter;
jsonConverter.setClazz(Object[].class);
recodeObject = recordConverter.byteToObject(decodeBytes);
datas = (Object[]) recodeObject;
}
schema = sourceDataEntry.getSchema();
entryType = sourceDataEntry.getEntryType();
queueName = sourceDataEntry.getQueueName();
timestamp = sourceDataEntry.getTimestamp();
}
DataEntryBuilder dataEntryBuilder = new DataEntryBuilder(schema);
dataEntryBuilder.entryType(entryType);
dataEntryBuilder.queue(queueName);
dataEntryBuilder.timestamp(timestamp);
List fields = schema.getFields();
if (null != fields && !fields.isEmpty()) {
for (Field field : fields) {
dataEntryBuilder.putFiled(field.getName(), datas[field.getIndex()]);
}
}
SinkDataEntry sinkDataEntry = dataEntryBuilder.buildSinkDataEntry(message.getQueueOffset());
return sinkDataEntry;
}
```
datas[0] = new String(message.getBody());
IMO, it will occur some problem in encode and decode.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating WorkerSinkTask.convertToSinkDataEntry and inspect the new String(message.getBody()) conversion in the default converter branch. Reproduce the reported encode/decode problem with a MessageExt body and verify that the resulting SinkDataEntry preserves the original message data.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.