JDBC history store create table DDL fails on MySQL and Oracle databases
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
## Bug report
**What Debezium connector do you use and what version?**
MySQL connector version 3.3.1.Final
---
**What is the connector configuration?**
Not really relevant for this issue, I left all relevant info out of the config
```
debezium:
format:
schemas:
enable: false
source:
schema:
history:
internal:
".": io.debezium.storage.jdbc.history.JdbcSchemaHistory
jdbc:
connection:
table:
ddl: "CREATE TABLE debezium_database_history (`id` VARCHAR(36) NOT NULL, `history_data` BLOB, `history_data_seq` INTEGER, `record_insert_ts` TIMESTAMP NOT NULL, `record_insert_seq` INTEGER NOT NULL, PRIMARY KEY (id, history_data_seq))"
store.only.captured.tables.ddl: true
poll.interval.ms: 25
offset:
flush.interval.ms: 0
storage:
".": io.debezium.storage.jdbc.offset.JdbcOffsetBackingStore
jdbc:
connection:
table:
ddl: "CREATE TABLE debezium_offset_storage (`id` VARCHAR(36) NOT NULL, `offset_key` VARCHAR(1255), `offset_val` VARCHAR(1255), `record_insert_ts` TIMESTAMP NOT NULL, `record_insert_seq` INTEGER NOT NULL, PRIMARY KEY (id))"
```
Important here is that we changed the datatype of history_data from VARCHAR(65000) to BLOB
---
**What is the captured database version and mode of deployment?**
on-prem, Percona cluster (https://docs.percona.com/percona-server/8.0/release-notes/8.0.41-32.html) with galera. Based on MySQL version 8.0.41. We have 3 nodes. 1 master and 2 slaves (which when no issues occur only replicated the data
---
**What behavior do you expect?**
Create table DDL should work on most well known database like MySQL, sqlLite, Oracle, ... for all typical encodings that are used
---
**What behavior do you see?**
When the default create table DDL is execute the first time:
```
CREATE TABLE debezium_database_history (`id` VARCHAR(36) NOT NULL, `history_data` VARCHAR(65000), `history_data_seq` INTEGER, `record_insert_ts` TIMESTAMP NOT NULL, `record_insert_seq` INTEGER NOT NULL, PRIMARY KEY (id))
```
The database will throw an error:
`[42000][1074] Column length too big for column 'history_data' (max = 16383); use BLOB or TEXT instead`
We should dynamically determine what kind of database we are using. We could switch to BLOB which is supported by MySQL, Oracle and SQLlite **but not for PostgreSQL** (they use bytea)
---
**Do you see the same behaviour using the latest released Debezium version?**
yes
**How to reproduce the issue using our [tutorial](https://github.com/debezium/debezium-examples/tree/main/tutorial) deployment?**
No reproducible since we do not use jdbc offset and history store in the tutorial
Contributor guide
Research direction
Start with the default CREATE TABLE DDL shown in the report and the configured JdbcSchemaHistory and JdbcOffsetBackingStore entry points. Compare the requested behavior across MySQL, SQLite, Oracle, and PostgreSQL, including the stated BLOB/bytea incompatibility; done means the table-creation DDL works for the relevant databases and encodings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql, postgresql, sqlite
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100