Aiven-Open / Aiven-Open/cloud-storage-connectors-for-apache-kafka
JSONL Source Connector Configuration
- Dominant language
- Java
- Stars
- 58
- Forks
- 39
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 5
Description
Hello!
I would like to publish messages out of a bucket using JSONL. It seems like the sink connector produces a message of the following format based on the documentation:
```
{
"key": ...,
"value": { ... }
}
```
I would like to publish Kafka messages with the `key` field being the Kafka Message Key, and the `value` field be the Kafka Message Value.
I am trying to have a file with multiple messages that I can then publish using this format, but I am running into issues because of the native key and (I'm assuming) schema-less parsing of JSON messages. Based on this line:
https://github.com/Aiven-Open/cloud-storage-connectors-for-apache-kafka/blob/main/commons/src/main/java/io/aiven/kafka/connect/common/source/input/TransformerFactory.java#L53
It seems like if JSONL is the `input.format`, the produced Kafka Connect Record will be without a schema. This leads to issues if you try to apply SMTs to the message to extract the `key` and `value` of the message using transforms like `ValueToKey` and `ExtractField` to use the `key` field as the message key, and using `ExtractField` on `value` to unwrap the JSON payload for the message value.
Is there another approach that would allow me to do this? Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.