redpanda-data / redpanda-data/console

Console strips unknown properties on /validate connectors call

Open
#1,485 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
4.3k
Forks
432
Avg merge
3d 6h
Merged PRs (30d)
40

Description

Use the following config for a connector (PostgreSQL Debezium):

{
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "topic.prefix": "p1",
    "database.hostname": "xxx.eu-west-1.rds.amazonaws.com",
    "database.user": "postgres",
    "database.password": "xxx",
    "database.dbname": "postgres",
    "database.sslmode": "disable",
    "plugin.name": "pgoutput",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "header.converter": "org.apache.kafka.connect.storage.SimpleHeaderConverter",
    "name": "debezium-postgresql-connector-1",
    "converters": "someConverter",
    "someConverter.type": "com.some.converter.debezium.SomeConverter",
    "someConverter.tables": "public.test_table",
    "someConverter.tables.public.test_table.columns": "tags,schedule"
}

The properties someConverter.* are dynamic based on the converters value. These properties are not send to Kafka Connect with the /validate call, causing the connector to be unable to valiidate the config, as the required converter properties (someConverter.type) are not given. The UI hangs and user cannot create the connector.

Expected behaviour:
All of the defined properties are sent with the /valdiate call.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the Console code that builds and sends the connector configuration to the Kafka Connect /validate endpoint. Reproduce the PostgreSQL Debezium configuration with the dynamic someConverter.* properties, then verify that all defined properties are included in the request and that the UI no longer hangs when validation requires them.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, typescript
Domain
api, frontend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.