airbytehq / airbytehq/airbyte

[source-mysql] Concurrent initial-load partitioning generates non-latin1 boundary strings for varchar PK on latin1 table → error 3988 "Conversion from collation utf8mb4_0900_ai_ci into latin1_swedish_ci impossible for parameter"

未關閉
#82,263 2 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
area/connectors autoteam community connectors/destination/snowflake connectors/source/mysql needs-triage team/extensibility type/bug
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

### Connector Name

source-mysql

### Connector Version

3.53.1

### What step the error happened?

During the sync

## Environment

- Airbyte OSS 1.6.0 (Helm deployment on EKS)
- source-mysql 3.53.1
- destination-snowflake 4.0.4
- MySQL 8.0.42 (Aurora RDS)
- Sync mode: Incremental | Append + Deduped, CDC (read changes using binary log)

## Relevant information

The initial snapshot of one specific table fails on every attempt (4/4) with:

```
java.sql.SQLException: Conversion from collation utf8mb4_0900_ai_ci into latin1_swedish_ci impossible for parameter
```

The failing table is large (~13.6 GB estimated) and has a **`varchar(128)` primary key on a table with `DEFAULT CHARSET=latin1`** (PK collation `latin1_swedish_ci`):

```sql
CREATE TABLE `person` (
`personID` varchar(128) NOT NULL,
-- ... ~45 more columns ...
PRIMARY KEY (`personID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
```

### Root cause analysis

Because the table exceeds the 3000 MiB target partition size, `MySqlJdbcConcurrentPartitionsCreator` splits the snapshot into 6 partitions:

```
i.a.i.s.m.MySqlJdbcConcurrentPartitionsCreator(run):521 Table memory size estimated at 13632 MiB.
i.a.i.s.m.MySqlJdbcConcurrentPartitionsCreator(run):522 Target partition size is 3000 MiB.
i.a.i.s.m.MySqlJdbcConcurrentPartitionsCreator(run):551 Table will be read by 6 concurrent partition reader(s).
```

Since the PK is a string, `MySqlSourceJdbcPartitionFactory.internalCalculateBoundaries` synthesizes midpoint boundary strings by interpolating between the sampled min/max keys. The real bounds are plain ASCII, but the four computed midpoints contain arbitrary Unicode code points, including supplementary-plane characters:

```
i.a.i.s.m.MySqlSourceJdbcPartitionFactory(internalCalculateBoundaries):702 boundaries:
[person7cry,
person?a𶛓򣎦]𶛐񬵃𶚫񬴼򣎠򣎨𶛓𶚯񬴼𶚽򣎓𶛐󙧽򣎈c𶚰򣎋𶚥𶚭򣎟5E_...,
personG_񬴵𶛓d񬴺󙨐񬴓󙨊𶛎𶛚...,
personO]򣎖󙨀k򣎥𶛝...,
personW[󙧸񬴭r󙨏򣎪...,
person_ZZZ]
```

These synthetic strings are then bound as prepared-statement parameters:

```
i.a.c.r.JdbcSelectQuerier$Result():79 Querying SELECT ... FROM `folkdata`.`person` WHERE (`personID` >= ?) AND (`personID` <= ?)
i.a.c.r.JdbcSelectQuerier$Result(initQueryExecution):103 Setting parameter #2 to Binding(value="person?a𶛓򣎦]𶛐...", type=StringFieldType)
```

The JDBC connection charset is utf8mb4, so MySQL 8 attempts to coerce the `utf8mb4_0900_ai_ci` parameter into the column's `latin1_swedish_ci` collation (to use the PK index) and fails with server error 3988, because the interpolated code points are not representable in latin1.

### Stack trace

```
java.sql.SQLException: Conversion from collation utf8mb4_0900_ai_ci into latin1_swedish_ci impossible for parameter
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:114)
at com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:612)
at com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:320)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1056)
at io.airbyte.cdk.read.JdbcSelectQuerier$Result.initQueryExecution(SelectQuerier.kt:107)
at io.airbyte.cdk.read.JdbcSelectQuerier$Result.(SelectQuerier.kt:81)
at io.airbyte.cdk.read.JdbcSelectQuerier.executeQuery(SelectQuerier.kt:58)
at io.airbyte.integrations.source.mysql.MySqlSourceSelectQuerier.executeQuery(MySqlSourceSelectQuerier.kt:31)
at io.airbyte.cdk.read.JdbcNonResumablePartitionReader.run(JdbcPartitionReader.kt:131)
at io.airbyte.cdk.read.FeedReader$readPartitionWithResources$4.invokeSuspend(FeedReader.kt:283)
...
```

`failureOrigin: source` on all 4 attempts; the retries can never succeed because the boundary interpolation deterministically produces non-latin1 code points.

### Conditions to reproduce

1. A table large enough to trigger the concurrent partition split (> ~3000 MiB estimated), AND
2. a string (varchar) primary key on a column whose charset is latin1 (or any charset smaller than utf8mb4), AND
3. PK values that are not GUID-shaped (so the GUID-specific handling from #69104 / #76050 does not apply — our keys look like `person`).

Setting connector concurrency to 1 does **not** avoid the bug — the log shows `Effective concurrency: 1` yet the table is still split into 6 partitions, and each partition query still uses the synthetic boundaries.

### Expected behavior

Partition boundary strings should be usable in comparisons against the PK column. Possible fixes:

- Clamp interpolated code points to the column's character set (for latin1, code points ≤ U+00FF), or
- use actual sampled key values as boundaries instead of synthetic midpoints (as source-mysql 3.11.x did), or
- wrap the parameter in `CONVERT(? USING )` / add an explicit `COLLATE` so the server can compare without coercing the parameter.

### Workaround

Downgrading the connection's source-mysql to 3.11.21 avoids the issue (its chunking uses real key values). This matches the workaround reported for the related partition-boundary regressions in #64909 / #64139.

### Related issues

- #64909 ([source-mysql] Regression in 3.50.x: ArithmeticException in partition boundary calculation)
- #64139 ([source-mysql] java.lang.ArithmeticException: / by zero)
- #69104 (Better partitioning for tables with GUID string primary key — does not cover non-GUID string PKs)

### Relevant log output

```shell
java.sql.SQLException: Conversion from collation utf8mb4_0900_ai_ci into latin1_swedish_ci impossible for parameter
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:114)
at com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:612)
at com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:320)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1056)
at io.airbyte.cdk.read.JdbcSelectQuerier$Result.initQueryExecution(SelectQuerier.kt:107)
at io.airbyte.cdk.read.JdbcSelectQuerier$Result.(SelectQuerier.kt:81)
at io.airbyte.cdk.read.JdbcSelectQuerier.executeQuery(SelectQuerier.kt:58)
at io.airbyte.integrations.source.mysql.MySqlSourceSelectQuerier.executeQuery(MySqlSourceSelectQuerier.kt:31)
at io.airbyte.cdk.read.JdbcNonResumablePartitionReader.run(JdbcPartitionReader.kt:131)
at io.airbyte.cdk.read.FeedReader$readPartitionWithResources$4.invokeSuspend(FeedReader.kt:283)
...
```

### Contribute

- [ ] Yes, I want to contribute

---
**Internal Tracking:** airbytehq/oncall#13104

貢獻指南

開啟貢獻指南

研究方向

Start at MySqlSourceJdbcPartitionFactory.internalCalculateBoundaries and trace its callers in MySqlJdbcConcurrentPartitionsCreator; then follow the query path through MySqlSourceSelectQuerier and JdbcSelectQuerier. Reproduce the large latin1 table case and verify that generated partition boundaries remain usable in comparisons without the MySQL collation error.

由索引模型根據 Issue 內容生成。

評估

技術堆疊
java, kotlin, mysql
領域
databases
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
冷清
描述清晰度
基本清楚
新手友好度
55/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。