Aiven-Open / Aiven-Open/cloud-storage-connectors-for-apache-kafka

JSONL Source Connector Configuration

Open
#558 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
58
Forks
39
Avg merge
2d 10h
Merged PRs (30d)
5

Description

Hello!

I would like to publish messages out of a bucket using JSONL. It seems like the sink connector produces a message of the following format based on the documentation:

```
{
"key": ...,
"value": { ... }
}
```

I would like to publish Kafka messages with the `key` field being the Kafka Message Key, and the `value` field be the Kafka Message Value.

I am trying to have a file with multiple messages that I can then publish using this format, but I am running into issues because of the native key and (I'm assuming) schema-less parsing of JSON messages. Based on this line:

https://github.com/Aiven-Open/cloud-storage-connectors-for-apache-kafka/blob/main/commons/src/main/java/io/aiven/kafka/connect/common/source/input/TransformerFactory.java#L53

It seems like if JSONL is the `input.format`, the produced Kafka Connect Record will be without a schema. This leads to issues if you try to apply SMTs to the message to extract the `key` and `value` of the message using transforms like `ValueToKey` and `ExtractField` to use the `key` field as the message key, and using `ExtractField` on `value` to unwrap the JSON payload for the message value.

Is there another approach that would allow me to do this? Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.