MySQL `REAL` columns emit FLOAT32 schema with silent precision loss when DDL is captured during streaming; snapshot emits FLOAT64 (default behavior inconsistent despite DBZ-6226)
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
## Bug report
**What Debezium connector do you use and what version?**
`io.debezium.connector.mysql.MySqlConnector`, reproduced on `3.0.7.Final`.
The responsible code paths are unchanged on current `main`:
- `MySqlAntlrDdlParser#initializeDataTypeResolver` maps the `REAL` DDL token to `java.sql.Types.REAL`
- `JdbcValueConverters#schemaBuilder` maps `Types.REAL` to `SchemaBuilder.float32()`
- `JdbcValueConverters#convertReal` narrows any incoming `Number` via `value.floatValue()`
---
**What is the connector configuration?**
```json
{
"name": "real-repro-source",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"database.hostname": "mysql-source",
"database.port": "3306",
"database.user": "mysqluser",
"database.password": "***",
"database.server.id": "5501",
"topic.prefix": "repro",
"database.include.list": "srcdb",
"table.include.list": "srcdb.t_real_.*",
"schema.history.internal.kafka.bootstrap.servers": "kafka:9092",
"schema.history.internal.kafka.topic": "repro.schema-history",
"snapshot.mode": "initial"
}
}
```
All type-mapping options are defaults (`decimal.handling.mode=precise`, `time.precision.mode=adaptive_time_microseconds`, no custom `converters`). Value converter: `JsonConverter` with schemas enabled.
---
**What is the captured database version and mode of deployment?**
MySQL 8.0.46 (Docker, `mysql:8.0`), default `sql_mode`
(`ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION` — i.e. `REAL_AS_FLOAT` is NOT enabled).
---
**What behavior do you expect?**
Per the MySQL manual ([Numeric Data Type Syntax](https://dev.mysql.com/doc/refman/8.4/en/numeric-type-syntax.html)), `REAL` is a synonym for `DOUBLE` unless the `REAL_AS_FLOAT` SQL mode is enabled. The server rewrites the column type at DDL time — `SHOW CREATE TABLE` reports `double` — stores 8-byte doubles, and binlog row events carry 8-byte doubles.
Therefore a `REAL` column should always be emitted as a `FLOAT64` schema with full double precision, regardless of whether the table definition was captured during snapshot or from a binlog DDL event, and the schema should be stable across the two phases.
---
**What behavior do you see?**
The same column definition produces different schemas and different values depending on how the table definition was captured:
| Capture path | Emitted schema | Emitted value for `3.141592653589793` |
|---|---|---|
| Snapshot (table existed before connector start) | `FLOAT64` (`"type":"double"`) | `3.141592653589793` (preserved) |
| Streaming (`CREATE TABLE ... REAL` while connector runs) | `FLOAT32` (`"type":"float"`) | `3.1415927` (**silently truncated**) |
The schema history topic shows why the two paths diverge:
```
snapshot: "CREATE TABLE `t_real_snap` (\n `id` int NOT NULL,\n `v` double DEFAULT NULL, ...)" <- SHOW CREATE TABLE output (server already normalized REAL -> double)
streaming: "CREATE TABLE t_real_stream (id INT PRIMARY KEY, v REAL)" <- original binlog statement (raw REAL keyword)
```
Root cause: the ANTLR DDL parser resolves the MySQL `REAL` keyword using its JDBC meaning (`java.sql.Types.REAL` = single precision) rather than its MySQL meaning (synonym for `DOUBLE`). `JdbcValueConverters#schemaBuilder` then legitimately maps `Types.REAL` to `float32()`, and `convertReal` narrows the 8-byte double coming from the binlog row event via `floatValue()`, corrupting the value.
Impact:
1. **Silent data corruption** — doubles truncated to ~7 significant digits for any `REAL` column whose table DDL was captured from the binlog.
2. **Schema instability** — the same table flips between `float`/`double` field schemas depending on capture path (e.g. after a re-snapshot), which can trip Schema Registry compatibility checks with Avro.
Relation to existing issues: this was reported as [DBZ-6226](https://redhat.atlassian.net/browse/DBZ-6226) (closed, fix version 2.2.0.Beta1). However, the shipped fix only introduced the **opt-in** `JdbcSinkDataTypesConverter` (commit `1c5f14f7f`); the default mapping was left unchanged, and DBZ-6226's own conclusion — "Regardless of when the table is created, REAL data types should be treated as a DOUBLE" — was never applied to the default path. The behavior above reproduces on 3.0.7.Final, two years after DBZ-6226 was closed.
Suggested fix: resolve the `REAL` token to `Types.DOUBLE` in `MySqlAntlrDdlParser#initializeDataTypeResolver`, matching both the server behavior and the JDBC driver metadata already used by the snapshot path. The MariaDB parser has the same `REAL` = `DOUBLE` semantics and likely needs the same change. (A `REAL_AS_FLOAT` caveat could be documented; the sql_mode is not currently inspected either way.) The documentation row `REAL[(M,D)] -> FLOAT32` in the shared MySQL/MariaDB data type mapping table should be updated accordingly.
Workaround for affected users:
```properties
converters=jdbc-sink
jdbc-sink.type=io.debezium.connector.binlog.converters.JdbcSinkDataTypesConverter
jdbc-sink.selector.real=srcdb\..*\.v
jdbc-sink.treat.real.as.double=true
```
---
**Do you see the same behaviour using the latest released Debezium version?**
Reproduced empirically on 3.0.7.Final. The responsible code (`MySqlAntlrDdlParser` REAL registration, `JdbcValueConverters` `Types.REAL` handling) is unchanged on current `main`, so the behavior is expected to be identical in the latest releases.
---
**Do you have the connector logs, ideally from start till finish?**
Not attached — the issue is deterministic and fully explained by the schema history + emitted messages above. Logs can be provided on request.
---
**How to reproduce the issue using our [tutorial](https://github.com/debezium/debezium-examples/tree/main/tutorial) deployment?**
1. Start the tutorial deployment (MySQL + Kafka + Connect).
2. Before registering the connector, create a table and probe row:
```sql
CREATE TABLE inventory.t_real_snap (id INT PRIMARY KEY, v REAL);
INSERT INTO inventory.t_real_snap VALUES (1, 3.141592653589793);
-- SHOW CREATE TABLE inventory.t_real_snap; -- note: column reported as `double`
```
3. Register the MySQL connector with `table.include.list=inventory.t_real_.*` and defaults otherwise.
4. Consume the snapshot topic — the `v` field schema is `double`, value preserved.
5. With the connector streaming, create a second table and insert the same value:
```sql
CREATE TABLE inventory.t_real_stream (id INT PRIMARY KEY, v REAL);
INSERT INTO inventory.t_real_stream VALUES (1, 3.141592653589793);
```
6. Consume the second topic — the `v` field schema is `float` and the value is `3.1415927`.
Contributor guide
Assessment
This issue has not been assessed yet.