debezium / debezium/dbz

JDBC history store create table DDL fails on MySQL and Oracle databases

Open
#13 1 comment 0 reactions 0 assignees View on GitHub
component/core-library type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

## Bug report

**What Debezium connector do you use and what version?**

MySQL connector version 3.3.1.Final

---

**What is the connector configuration?**

Not really relevant for this issue, I left all relevant info out of the config

```
debezium:
format:
schemas:
enable: false
source:
schema:
history:
internal:
".": io.debezium.storage.jdbc.history.JdbcSchemaHistory
jdbc:
connection:
table:
ddl: "CREATE TABLE debezium_database_history (`id` VARCHAR(36) NOT NULL, `history_data` BLOB, `history_data_seq` INTEGER, `record_insert_ts` TIMESTAMP NOT NULL, `record_insert_seq` INTEGER NOT NULL, PRIMARY KEY (id, history_data_seq))"
store.only.captured.tables.ddl: true
poll.interval.ms: 25
offset:
flush.interval.ms: 0
storage:
".": io.debezium.storage.jdbc.offset.JdbcOffsetBackingStore
jdbc:
connection:
table:
ddl: "CREATE TABLE debezium_offset_storage (`id` VARCHAR(36) NOT NULL, `offset_key` VARCHAR(1255), `offset_val` VARCHAR(1255), `record_insert_ts` TIMESTAMP NOT NULL, `record_insert_seq` INTEGER NOT NULL, PRIMARY KEY (id))"
```

Important here is that we changed the datatype of history_data from VARCHAR(65000) to BLOB

---

**What is the captured database version and mode of deployment?**

on-prem, Percona cluster (https://docs.percona.com/percona-server/8.0/release-notes/8.0.41-32.html) with galera. Based on MySQL version 8.0.41. We have 3 nodes. 1 master and 2 slaves (which when no issues occur only replicated the data

---

**What behavior do you expect?**

Create table DDL should work on most well known database like MySQL, sqlLite, Oracle, ... for all typical encodings that are used

---

**What behavior do you see?**

When the default create table DDL is execute the first time:

```
CREATE TABLE debezium_database_history (`id` VARCHAR(36) NOT NULL, `history_data` VARCHAR(65000), `history_data_seq` INTEGER, `record_insert_ts` TIMESTAMP NOT NULL, `record_insert_seq` INTEGER NOT NULL, PRIMARY KEY (id))
```

The database will throw an error:

`[42000][1074] Column length too big for column 'history_data' (max = 16383); use BLOB or TEXT instead`

We should dynamically determine what kind of database we are using. We could switch to BLOB which is supported by MySQL, Oracle and SQLlite **but not for PostgreSQL** (they use bytea)

---

**Do you see the same behaviour using the latest released Debezium version?**

yes

**How to reproduce the issue using our [tutorial](https://github.com/debezium/debezium-examples/tree/main/tutorial) deployment?**

No reproducible since we do not use jdbc offset and history store in the tutorial

Contributor guide

Open the contributing guide

Research direction

Start with the default CREATE TABLE DDL shown in the report and the configured JdbcSchemaHistory and JdbcOffsetBackingStore entry points. Compare the requested behavior across MySQL, SQLite, Oracle, and PostgreSQL, including the stated BLOB/bytea incompatibility; done means the table-creation DDL works for the relevant databases and encodings.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql, postgresql, sqlite
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.