redpanda-data / redpanda-data/connect
sql_select: Validate ClickHouse data types
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
Internally, Bloblang numbers are JSON numbers. Connect passes them to the ClickHouse driver as strings by default. When using the ClickHouse driver, you nearly always need to explicitly cast integers to sized types using methods like int64().
We encountered a footgun related to this. One of our pipelines queries ClickHouse to see if a record exists like so:
SELECT id FROM tbl WHERE id = ?
Because we forgot to cast the id Bloblang value to an int64(), ClickHouse actually executed:
SELECT id FROM tbl WHERE id = '12345'
This type mismatch causes ClickHouse to scan the table instead of using the primary key.
The sql_select processor could support a validate_column_types field to validate incoming args_mapping values match the target data types.
sql_select already accepts a table name and column names. Connect could inspect the table when it starts, or when the processor is used for the first time, to determine the column data types. With validate_column_types: true, sql_select would validate incoming data against the detected data types.
The validate_column_types field could also control whether Connect inspects the target table in the first place.
Initially, this field would only affect the ClickHouse output.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the sql_select processor, its table and column handling, and the args_mapping path. Determine how ClickHouse target types could be inspected and how validate_column_types should affect validation; done means mismatched values are rejected for ClickHouse without changing the default behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, go
- Domain
- database
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100