airbytehq / airbytehq/airbyte

[source-mysql] Concurrent initial snapshot drops every row sharing the minimum first primary-key column value on composite-PK tables

未关闭
#85,351 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
autoteam community connectors/source/mssql connectors/source/mysql connectors/source/postgres hyd-review team/use
主要语言
Python
星标
22.1k
派生
5.3k
PR 合并指标
PR 指标待抓取

描述

### Connector Name

source-mysql

### Connector Version

3.53.4 (also reproduced on 3.53.0 - 3.53.3; code unchanged on master)

### What step the error happened?

During the sync

### Relevant information

On a full initial snapshot (CDC incremental, concurrent mode) of a table with a **composite primary key** that is large enough to be split into concurrent partitions, every row sharing the minimum value of the first primary-key column is silently skipped. The sync succeeds and no error is logged; the destination simply lacks the `MIN(first pk column)` group.

Cause: concurrent partitions are bounded on the first primary-key column only, so a split partition's lower bound is a strict prefix of the checkpoint columns. In `MySqlSourceJdbcPartition.where` the inclusive comparison is gated on `isLowerBoundIncluded && idx == checkpointColumns.size - 1`, which can never be true for a prefix bound on a composite key, so the first partition is read as `pk0 > min` instead of `pk0 >= min`. Single-column primary keys get `pk0 >= min` and are unaffected. The same predicate exists in the Bulk CDK (`buildWhereClause` in extract-jdbc, `TriggerPartition` in extract-trigger) and in source-postgres / source-mssql.

Observed on MySQL 8.4 -> Snowflake after a connection clear. Every partitioned table with a composite key lost exactly its `MIN(first pk column)` group (exact counts against the source), across seven tables:

| composite PK shape | rows dropped |
|---|---|
| (varchar, varchar, varchar, varchar, datetime) | 15,540 |
| (varchar, varchar, varchar, varchar) | 14,585 |
| (varchar, varchar, varchar, varchar) | 14,585 |
| (varchar, varchar, varchar, varchar) | 9,205 |
| (varchar, varchar, varchar, varchar, datetime) | 3,790 |
| (varchar, varchar, varchar, varchar, datetime) | 955 |
| (int, int) | 1 |

Steps to reproduce: create `t(id INT, sub INT, payload VARCHAR(255), PRIMARY KEY(id, sub))`, insert 300 x 100 rows, make it large enough to be partitioned (or lower `airbyte.connector.extract.jdbc.expected-throughput-bytes-per-second`), run a CDC initial snapshot: 29,900 of 30,000 rows arrive, all rows with `id = 1` are missing.

### Relevant log output

```text
Every first partition query in the job is issued with an exclusive lower bound; `>=` never appears:
SELECT ... FROM `db`.`table_a` WHERE (`pk0` > ?) AND (`pk0` <= ?)
SELECT ... FROM `db`.`table_b` WHERE (`pk0` > ?) AND (`pk0` <= ?)
```

### Contribute

- [x] Yes, I want to contribute

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。