[Feature][Connectors-v2][Kafka][Source] Cannot extract data from both metadata and value simultaneously
- Dominant language
- Java
- Stars
- 9.7k
- Forks
- 2.4k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 203
Description
### Search before asking
- [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
When using Kafka connector to read data from Kafka, if `format`="NATIVE", metadata information and a value of byte [] type are returned. But there are some scenarios, such as users wanting to retrieve both metadata and a certain field value in the value. The `value` data is as follows:
```
{
"aa":123,
"bb":"testStr"
}
```
And the metadata is as follows:
```
{
"headers": {
"header1": "header1",
"header2": "header2"
},
"key": "dGVzdF9ieXRlc19kYXRh",
"partition": 3,
"timestamp": 1672531200000,
"timestampType": "CREATE_TIME",
}
```
If the user wants to obtain the values of timestamp and aa at the same time, seatunnel cannot meet the requirements.
I also learned that Dataworks can support this feature:
Can we implement this feature? I want to try it out.My idea is to add a `valueJson` value and return the following message
```
{
"headers": {
"header1": "header1",
"header2": "header2"
},
"key": "dGVzdF9ieXRlc19kYXRh",
"partition": 3,
"timestamp": 1672531200000,
"timestampType": "CREATE_TIME",
"value": "dGVzdF9ieXRlc19kYXRh",
"valueJson": {
"aa": 123,
"bb": "testStr"
}
}
```
### Usage Scenario
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the Kafka source connector's handling of format="NATIVE", focusing on how metadata and the byte[] value are currently returned. Trace how a JSON value could be exposed alongside metadata and confirm the resulting record supports retrieving fields such as timestamp and aa together, matching the example output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100