debezium / debezium/dbz

OpenLineage config not working with composite, and filter out debezium_config openlineage kafka message for security reasons [DBZ-9555]

Open
#1,414 0 comments 0 reactions 0 assignees View on GitHub
component/core-library migrated-from-jira type/bug
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-9555](https://issues.redhat.com/browse/DBZ-9555)

In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.
h1. Bug report

For bug reports, provide this information, please:
h2. What Debezium connector do you use and what version?

3.3.0 Final
h2. What is the connector configuration?
class: io.debezium.connector.postgresql.PostgresConnector
  tasksMax: 1
  # https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-required-configuration-properties
  config:
    #################
    # debezium config
    #################
    database.hostname: "mypostgre" # hostname of secondary
    database.port: "32521" # secondary port
    database.user: northwind # username with login and replication rights
    database.password: mypassword # password
    database.dbname: northwind # name of db to sync
    plugin.name: pgoutput
    slot.name: debezium_replication_slot
    slot.failover: true
    slot.drop.on.stop: false
    publication.name: debezium_publication
    publication.autocreate.mode: disabled

    # slot.initial.actions: none
   
    # prefix for generated kafka topics
    topic.prefix: dev.poc_postgres.northwind
   
    # list of schemas to fetch, comma separated
    schema.include.list: public
    # list of tables to fetch, comma separated
    #table.include.list: public., public.
   
    #################
    # kafka connect config
    #################
   
    key.converter: org.apache.kafka.connect.json.JsonConverter
    #key.converter: org.apache.kafka.connect.storage.StringConverter #io.confluent.connect.avro.AvroConverter
    # serializer config - how to serialize data, possible values: json, avro, protobuf, string, bytearray, etc.
    value.converter: io.confluent.connect.avro.AvroConverter
    #value.converter: org.apache.kafka.connect.json.JsonConverter
   
    value.converter.schema.registry.url: http://apicurio-schema-registry-app-service:8080/apis/ccompat/v7

    schema.change.data.capture.event.logging.enabled: "true"
    schema.history.internal.kafka.bootstrap.servers: "kafka-dev-broker-plain-0:9092,kafka-dev-broker-plain-1:9092,kafka-dev-broker-plain-2:9092"
    schema.history.internal.kafka.topic: "__debezium_postgres_poc_northwind"
   
    # Shows schema information in message (true for verbose))
    key.converter.schemas.enable: "false"
    value.converter.schemas.enable: "false"
   
    # Optional: Einstellungen für Kafka Connect, um Deletes als Tombstones zu behandeln
    delete.handling.mode: rewrite

    slot.max.retries: "10" # Optional: Mehr Versuche
    slot.retry.delay.ms: "10000" # Optional: längere Wartezeit zwischen Versuchen (10s)

    # exactly once support, not bound to debezium but for kafka connect
    exactly.once.support: required

    openlineage.integration.enabled: true
    # does not work with composite
    #openlineage.integration.config.file.path: "/mnt/openlineage/marquez-connect-config.yml"
h2. What is the captured database version and mode of deployment?

postgres 18, k8s cloudnativepg operator
h2. What behavior do you expect?

composite config is being used correctly, maybe also bug in openlineage itself

debezium does not send its configuration into kafka
h2. What behavior do you see?

composite problem:

does not work. needs to be set by env variables in order to work like:
env:
        - name: OPENLINEAGE__TRANSPORT__TYPE
          value: "composite"
        - name: OPENLINEAGE__TRANSPORT__CONTINUE_ON_FAILURE
          value: "true"
        - name: OPENLINEAGE__TRANSPORT__TRANSPORTS__MARQUEZ__TYPE
          value: "http"
        - name: OPENLINEAGE__TRANSPORT__TRANSPORTS__MARQUEZ__URL
          value: "http://marquez:5000"
        - name: OPENLINEAGE__TRANSPORT__TRANSPORTS__MARQUEZ__ENDPOINT
          value: "api/v1/lineage"
        - name: OPENLINEAGE__TRANSPORT__TRANSPORTS__MARQUEZ__COMPRESSION
          value: "gzip"
        - name: OPENLINEAGE__TRANSPORT__TRANSPORTS__KAFKA__TYPE
          value: "kafka"
        - name: OPENLINEAGE__TRANSPORT__TRANSPORTS__KAFKA__TOPIC_NAME
          value: "dev.openlineage.postgres.debezium"
        - name: OPENLINEAGE__TRANSPORT__TRANSPORTS__KAFKA__PROPERTIES
          value: "mykafka:9090"
h2. Do you see the same behaviour using the latest released Debezium version?

yes 3.3.0 final
h2. Do you have the connector logs, ideally from start till finish?

Yes but logs are all fine
h2. How to reproduce the issue using our [tutorial|https://github.com/debezium/debezium-examples/tree/main/tutorial] deployment?

look at the created kafka messages from debezium from openlineage. they send the whole debezium config with it, so with db credentials etc.
h1. Feature request or enhancement

do not send config of debezium into openlineage kafka topci
h2. Which use case/requirement will be addressed by the proposed feature?

security
h2. Implementation ideas (optional)

openlineage configuration

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.