debezium / debezium/dbz

Output format for value and key for debeziumserver CR

Open
#1,963 7 comments 2 reactions 0 assignees View on GitHub
component/debezium-platform type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

## Feature request or enhancement

_For feature requests or enhancements, please provide the following information:_

**Which use case/requirement will be addressed by the proposed feature?**

`After integrating DMP v3.4.0 with Apicurio Schema registry, I found an issue where debezium.format.key and debezium.format.value are by default json. I want to change it to avro, but found no option to do it.

Looks like the format key and value are now taken Json as default from conductor source code. I need to be able to pass it dynamically. I can see debeziumserver CRD, but couldn't find an option to pass the values, through helm or through pipeline api calls.

I tried passing through source payload, but it is not working as intended.

Here is my source payload

"config": {
"slot.name": "synic_tab_test_slot",
"tasks.max": "1",
"plugin.name": "pgoutput",
"topic.prefix": "pg17_test",
"database.port": "5432",
"database.user": "user",
"format.key": "avro",
"format.value": "avro",
"key.converter": "io.apicurio.registry.utils.converter.AvroConverter",
"snapshot.mode": "when_needed",
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.dbname": "dbname",
"value.converter": "io.apicurio.registry.utils.converter.AvroConverter",
"publication.name": "synic_tab_test_pub",
"database.hostname": "hostname",
"database.password": "password",
"table.include.list": "synic_tab.table",
"schema.include.list": "synic_tab",
"decimal.handling.mode": "double",
"heartbeat.interval.ms": "10000",
"heartbeat.action.query": "SELECT pg_logical_emit_message(false, 'heartbeat', now()::varchar)",
"publication.autocreate.mode": "disabled",
"schema.name.adjustment.mode": "avro",
"producer.override.compression.type": "lz4",
"key.converter.apicurio.registry.url": "http://apicurioregistry-blue-app-service.wms.svc.cluster.local:8080/",
"key.converter.apicurio.auth.client.id": "apicurio-registry-app",
"value.converter.apicurio.registry.url": "http://apicurioregistry-blue-app-service.wms.svc.cluster.local:8080/",
"value.converter.apicurio.auth.client.id": "apicurio-registry-app",
"key.converter.apicurio.auth.client.secret": "secret",
"key.converter.apicurio.registry.find-latest": "true",
"value.converter.apicurio.auth.client.secret": "secret",
"key.converter.apicurio.registry.as-confluent": "true",
"key.converter.apicurio.registry.auto-register": "true",
"value.converter.apicurio.registry.find-latest": "true",
"value.converter.apicurio.registry.as-confluent": "true",
"key.converter.apicurio.registry.headers.enabled": "false",
"value.converter.apicurio.registry.auto-register": "true",
"value.converter.apicurio.registry.headers.enabled": "false",
"key.converter.apicurio.auth.service.token.endpoint": "http://keycloak/realms/sso/protocol/openid-connect/token",
"value.converter.apicurio.auth.service.token.endpoint": "http://keycloak/realms/sso/protocol/openid-connect/token"
},
"connection": {
"id": 19,
},
"description": null,
"name": "pg18-synic-tab-test-source",
"schema": "postgresql",
"type": "io.debezium.connector.postgresql.PostgresConnector",
"vaults": []
}

It comes are the below image in [application.properties](http://application.properties/). The default settings still remain the same and the formats passed from source are not honored

Image

`

---

**Implementation ideas (optional)**

`Should be able to change the format.value and format.key dynamically. `

Contributor guide

Open the contributing guide

Research direction

Start at the debeziumserver CRD and trace how the source payload is converted into application.properties, including the conductor configuration path and Helm or pipeline API handling. Verify where debezium.format.key and debezium.format.value receive their defaults and identify the relevant configuration tests or entry points. Done means dynamically supplied key and value formats are preserved instead of being replaced by JSON defaults.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, postgresql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.