googleapis / googleapis/java-pubsub-group-kafka-connector

Support copying of JSON data to pubsub topic as JSON instead of stringified map.

Open
#202 5 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

priority: p3 type: feature request
Dominant language
Java
Stars
50
Forks
37
Avg merge
1d 19h
Merged PRs (30d)
3

Description

Problem: The issue is with the sink connector, If schema is not used, JSON data in topic is pushed to pubsub topic as string value of the map.
We need a feature so that we can force the output to be JSON and could be interpreted as JSON in pubsub.

Example for JSON object
{"name":"bob","id":45}

it is shown as on pubsub topic
{name=bob, id=45}

Add a configuration which could be used to force output to be JSON.

Alternativies considered:
Tried forcing the output to be string using configuration
value.converter=org.apache.kafka.connect.storage.StringConverter

pubsub topic gets the right value but the single message transforms didn't not work, for example if you want to copy json field to header, it does not work.

Example: Sample SMT

"headers.publish" : true,
"transforms": "insertAppIdHeader,moveFieldsToHeader",
"transforms.moveFieldsToHeader.type": "org.apache.kafka.connect.transforms.HeaderFrom$Value",
"transforms.moveFieldsToHeader.fields": "name,id",
"transforms.moveFieldsToHeader.headers": "name,id",

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.