googleapis / googleapis/java-pubsub-group-kafka-connector
Support copying of JSON data to pubsub topic as JSON instead of stringified map.
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 50
- Forks
- 37
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 3
Description
Problem: The issue is with the sink connector, If schema is not used, JSON data in topic is pushed to pubsub topic as string value of the map.
We need a feature so that we can force the output to be JSON and could be interpreted as JSON in pubsub.
Example for JSON object
{"name":"bob","id":45}
it is shown as on pubsub topic
{name=bob, id=45}
Add a configuration which could be used to force output to be JSON.
Alternativies considered:
Tried forcing the output to be string using configuration
value.converter=org.apache.kafka.connect.storage.StringConverter
pubsub topic gets the right value but the single message transforms didn't not work, for example if you want to copy json field to header, it does not work.
Example: Sample SMT
"headers.publish" : true,
"transforms": "insertAppIdHeader,moveFieldsToHeader",
"transforms.moveFieldsToHeader.type": "org.apache.kafka.connect.transforms.HeaderFrom$Value",
"transforms.moveFieldsToHeader.fields": "name,id",
"transforms.moveFieldsToHeader.headers": "name,id",
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.