MySQL Source Connector: map columns declared as BOOLEAN to INT16 by default, consistently with the snapshot phase
- Dominant language
- HTML
- Stars
- 6
- Forks
- 9
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
## Feature request or enhancement
A column declared as `BOOLEAN` is mapped differently depending on how its schema reaches the connector:
- During snapshots the schema comes from `SHOW CREATE TABLE`, which returns `tinyint(1)`, so the field is emitted as `INT16` with values `0`/`1`.
- A `CREATE`/`ALTER` statement parsed during streaming carries the original `BOOLEAN` keyword, so the same declaration produces a Kafka `boolean` field with `true`/`false` (`MySqlAntlrDdlParser` maps `BOOL`/`BOOLEAN` to
`Types.BOOLEAN`, which `JdbcValueConverters` turns into `SchemaBuilder.bool()`).
Both representations survive restarts, because the schema history replays each table's original text, so a single pipeline can permanently contain both encodings for identically declared columns. The behavior is described
in the "Boolean values" section of the documentation, with the `TinyIntOneToBooleanConverter` as the workaround.
I would like to propose making `INT16` the default for the streaming path as well:
- Consistency is only achievable in this direction: `SHOW CREATE TABLE` cannot distinguish `BOOLEAN` from `TINYINT(1)`, and the `TINYINT(1)` length heuristic no longer works for e.g. `TINYINT(1) UNSIGNED` on MySQL 8.0.19+.
- MySQL itself rewrites `BOOLEAN` to `TINYINT(1)` and does not enforce boolean semantics (the column can store any value from -128 to 127); the streaming path silently coerces such values to `true` today without the user
opting in.
- Users who want boolean semantics already have the opt-in `TinyIntOneToBooleanConverter` (DBZ-1801), which applies uniformly to both phases.
Since this changes the emitted schema for streaming-created `BOOLEAN` columns (`boolean` -> `int16`), it would have to be gated on a major release or introduced behind an option. I am happy to work on a PR if the direction
is agreed on.
Contributor guide
Research direction
Start with the Boolean values documentation section and compare MySqlAntlrDdlParser's BOOLEAN mapping with JdbcValueConverters and the TinyIntOneToBooleanConverter. Determine whether the INT16 default should be gated by a major release or an option; the work is done when the agreed behavior is consistently defined for snapshot and streaming schemas and documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100