redpanda-data / redpanda-data/connect

sql_select: Validate ClickHouse data types

Open
#4,192 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

Internally, Bloblang numbers are JSON numbers. Connect passes them to the ClickHouse driver as strings by default. When using the ClickHouse driver, you nearly always need to explicitly cast integers to sized types using methods like int64().

We encountered a footgun related to this. One of our pipelines queries ClickHouse to see if a record exists like so:

SELECT id FROM tbl WHERE id = ?

Because we forgot to cast the id Bloblang value to an int64(), ClickHouse actually executed:

SELECT id FROM tbl WHERE id = '12345'

This type mismatch causes ClickHouse to scan the table instead of using the primary key.


The sql_select processor could support a validate_column_types field to validate incoming args_mapping values match the target data types.

sql_select already accepts a table name and column names. Connect could inspect the table when it starts, or when the processor is used for the first time, to determine the column data types. With validate_column_types: true, sql_select would validate incoming data against the detected data types.

The validate_column_types field could also control whether Connect inspects the target table in the first place.

Initially, this field would only affect the ClickHouse output.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the sql_select processor, its table and column handling, and the args_mapping path. Determine how ClickHouse target types could be inspected and how validate_column_types should affect validation; done means mismatched values are rejected for ClickHouse without changing the default behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, go
Domain
database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.