apache / apache/seatunnel

[Feature][Connectors-v2][Kafka][Source] Cannot extract data from both metadata and value simultaneously

Open
#10,045 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 13h
Merged PRs (30d)
203

Description

### Search before asking

- [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.

### Description

When using Kafka connector to read data from Kafka, if `format`="NATIVE", metadata information and a value of byte [] type are returned. But there are some scenarios, such as users wanting to retrieve both metadata and a certain field value in the value. The `value` data is as follows:
```
{
"aa":123,
"bb":"testStr"
}
```
And the metadata is as follows:
```
{
"headers": {
"header1": "header1",
"header2": "header2"
},
"key": "dGVzdF9ieXRlc19kYXRh",
"partition": 3,
"timestamp": 1672531200000,
"timestampType": "CREATE_TIME",
}
```

If the user wants to obtain the values of timestamp and aa at the same time, seatunnel cannot meet the requirements.
I also learned that Dataworks can support this feature:

Image

Can we implement this feature? I want to try it out.My idea is to add a `valueJson` value and return the following message
```
{
"headers": {
"header1": "header1",
"header2": "header2"
},
"key": "dGVzdF9ieXRlc19kYXRh",
"partition": 3,
"timestamp": 1672531200000,
"timestampType": "CREATE_TIME",
"value": "dGVzdF9ieXRlc19kYXRh",
"valueJson": {
"aa": 123,
"bb": "testStr"
}
}

```
### Usage Scenario

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the Kafka source connector's handling of format="NATIVE", focusing on how metadata and the byte[] value are currently returned. Trace how a JSON value could be exposed alongside metadata and confirm the resulting record supports retrieving fields such as timestamp and aa together, matching the example output.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.