[source-mysql] Concurrent initial snapshot drops every row sharing the minimum first primary-key column value on composite-PK tables
- 主要语言
- Python
- 星标
- 22.1k
- 派生
- 5.3k
- PR 合并指标
- PR 指标待抓取
描述
### Connector Name
source-mysql
### Connector Version
3.53.4 (also reproduced on 3.53.0 - 3.53.3; code unchanged on master)
### What step the error happened?
During the sync
### Relevant information
On a full initial snapshot (CDC incremental, concurrent mode) of a table with a **composite primary key** that is large enough to be split into concurrent partitions, every row sharing the minimum value of the first primary-key column is silently skipped. The sync succeeds and no error is logged; the destination simply lacks the `MIN(first pk column)` group.
Cause: concurrent partitions are bounded on the first primary-key column only, so a split partition's lower bound is a strict prefix of the checkpoint columns. In `MySqlSourceJdbcPartition.where` the inclusive comparison is gated on `isLowerBoundIncluded && idx == checkpointColumns.size - 1`, which can never be true for a prefix bound on a composite key, so the first partition is read as `pk0 > min` instead of `pk0 >= min`. Single-column primary keys get `pk0 >= min` and are unaffected. The same predicate exists in the Bulk CDK (`buildWhereClause` in extract-jdbc, `TriggerPartition` in extract-trigger) and in source-postgres / source-mssql.
Observed on MySQL 8.4 -> Snowflake after a connection clear. Every partitioned table with a composite key lost exactly its `MIN(first pk column)` group (exact counts against the source), across seven tables:
| composite PK shape | rows dropped |
|---|---|
| (varchar, varchar, varchar, varchar, datetime) | 15,540 |
| (varchar, varchar, varchar, varchar) | 14,585 |
| (varchar, varchar, varchar, varchar) | 14,585 |
| (varchar, varchar, varchar, varchar) | 9,205 |
| (varchar, varchar, varchar, varchar, datetime) | 3,790 |
| (varchar, varchar, varchar, varchar, datetime) | 955 |
| (int, int) | 1 |
Steps to reproduce: create `t(id INT, sub INT, payload VARCHAR(255), PRIMARY KEY(id, sub))`, insert 300 x 100 rows, make it large enough to be partitioned (or lower `airbyte.connector.extract.jdbc.expected-throughput-bytes-per-second`), run a CDC initial snapshot: 29,900 of 30,000 rows arrive, all rows with `id = 1` are missing.
### Relevant log output
```text
Every first partition query in the job is issued with an exclusive lower bound; `>=` never appears:
SELECT ... FROM `db`.`table_a` WHERE (`pk0` > ?) AND (`pk0` <= ?)
SELECT ... FROM `db`.`table_b` WHERE (`pk0` > ?) AND (`pk0` <= ?)
```
### Contribute
- [x] Yes, I want to contribute
贡献指南
评估
这个 Issue 还没有评估数据。