Output format for value and key for debeziumserver CR
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
## Feature request or enhancement
_For feature requests or enhancements, please provide the following information:_
**Which use case/requirement will be addressed by the proposed feature?**
`After integrating DMP v3.4.0 with Apicurio Schema registry, I found an issue where debezium.format.key and debezium.format.value are by default json. I want to change it to avro, but found no option to do it.
Looks like the format key and value are now taken Json as default from conductor source code. I need to be able to pass it dynamically. I can see debeziumserver CRD, but couldn't find an option to pass the values, through helm or through pipeline api calls.
I tried passing through source payload, but it is not working as intended.
Here is my source payload
"config": {
"slot.name": "synic_tab_test_slot",
"tasks.max": "1",
"plugin.name": "pgoutput",
"topic.prefix": "pg17_test",
"database.port": "5432",
"database.user": "user",
"format.key": "avro",
"format.value": "avro",
"key.converter": "io.apicurio.registry.utils.converter.AvroConverter",
"snapshot.mode": "when_needed",
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.dbname": "dbname",
"value.converter": "io.apicurio.registry.utils.converter.AvroConverter",
"publication.name": "synic_tab_test_pub",
"database.hostname": "hostname",
"database.password": "password",
"table.include.list": "synic_tab.table",
"schema.include.list": "synic_tab",
"decimal.handling.mode": "double",
"heartbeat.interval.ms": "10000",
"heartbeat.action.query": "SELECT pg_logical_emit_message(false, 'heartbeat', now()::varchar)",
"publication.autocreate.mode": "disabled",
"schema.name.adjustment.mode": "avro",
"producer.override.compression.type": "lz4",
"key.converter.apicurio.registry.url": "http://apicurioregistry-blue-app-service.wms.svc.cluster.local:8080/",
"key.converter.apicurio.auth.client.id": "apicurio-registry-app",
"value.converter.apicurio.registry.url": "http://apicurioregistry-blue-app-service.wms.svc.cluster.local:8080/",
"value.converter.apicurio.auth.client.id": "apicurio-registry-app",
"key.converter.apicurio.auth.client.secret": "secret",
"key.converter.apicurio.registry.find-latest": "true",
"value.converter.apicurio.auth.client.secret": "secret",
"key.converter.apicurio.registry.as-confluent": "true",
"key.converter.apicurio.registry.auto-register": "true",
"value.converter.apicurio.registry.find-latest": "true",
"value.converter.apicurio.registry.as-confluent": "true",
"key.converter.apicurio.registry.headers.enabled": "false",
"value.converter.apicurio.registry.auto-register": "true",
"value.converter.apicurio.registry.headers.enabled": "false",
"key.converter.apicurio.auth.service.token.endpoint": "http://keycloak/realms/sso/protocol/openid-connect/token",
"value.converter.apicurio.auth.service.token.endpoint": "http://keycloak/realms/sso/protocol/openid-connect/token"
},
"connection": {
"id": 19,
},
"description": null,
"name": "pg18-synic-tab-test-source",
"schema": "postgresql",
"type": "io.debezium.connector.postgresql.PostgresConnector",
"vaults": []
}
It comes are the below image in [application.properties](http://application.properties/). The default settings still remain the same and the formats passed from source are not honored
`
---
**Implementation ideas (optional)**
`Should be able to change the format.value and format.key dynamically. `
Contributor guide
Research direction
Start at the debeziumserver CRD and trace how the source payload is converted into application.properties, including the conductor configuration path and Helm or pipeline API handling. Verify where debezium.format.key and debezium.format.value receive their defaults and identify the relevant configuration tests or entry points. Done means dynamically supplied key and value formats are preserved instead of being replaced by JSON defaults.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, postgresql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100