airbytehq / airbytehq/airbyte

[source-mysql] Concurrent initial-load partitioning generates non-latin1 boundary strings for varchar PK on latin1 table → error 3988 "Conversion from collation utf8mb4_0900_ai_ci into latin1_swedish_ci impossible for parameter"

Đang mở
#82,263 2 bình luận 0 reaction 0 người được giao Xem trên GitHub
area/connectors autoteam community connectors/destination/snowflake connectors/source/mysql needs-triage team/extensibility type/bug
Ngôn ngữ chính
Python
Star
22.1k
Fork
5.3k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

### Connector Name

source-mysql

### Connector Version

3.53.1

### What step the error happened?

During the sync

## Environment

- Airbyte OSS 1.6.0 (Helm deployment on EKS)
- source-mysql 3.53.1
- destination-snowflake 4.0.4
- MySQL 8.0.42 (Aurora RDS)
- Sync mode: Incremental | Append + Deduped, CDC (read changes using binary log)

## Relevant information

The initial snapshot of one specific table fails on every attempt (4/4) with:

```
java.sql.SQLException: Conversion from collation utf8mb4_0900_ai_ci into latin1_swedish_ci impossible for parameter
```

The failing table is large (~13.6 GB estimated) and has a **`varchar(128)` primary key on a table with `DEFAULT CHARSET=latin1`** (PK collation `latin1_swedish_ci`):

```sql
CREATE TABLE `person` (
`personID` varchar(128) NOT NULL,
-- ... ~45 more columns ...
PRIMARY KEY (`personID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
```

### Root cause analysis

Because the table exceeds the 3000 MiB target partition size, `MySqlJdbcConcurrentPartitionsCreator` splits the snapshot into 6 partitions:

```
i.a.i.s.m.MySqlJdbcConcurrentPartitionsCreator(run):521 Table memory size estimated at 13632 MiB.
i.a.i.s.m.MySqlJdbcConcurrentPartitionsCreator(run):522 Target partition size is 3000 MiB.
i.a.i.s.m.MySqlJdbcConcurrentPartitionsCreator(run):551 Table will be read by 6 concurrent partition reader(s).
```

Since the PK is a string, `MySqlSourceJdbcPartitionFactory.internalCalculateBoundaries` synthesizes midpoint boundary strings by interpolating between the sampled min/max keys. The real bounds are plain ASCII, but the four computed midpoints contain arbitrary Unicode code points, including supplementary-plane characters:

```
i.a.i.s.m.MySqlSourceJdbcPartitionFactory(internalCalculateBoundaries):702 boundaries:
[person7cry,
person?a𶛓򣎦]𶛐񬵃𶚫񬴼򣎠򣎨𶛓𶚯񬴼𶚽򣎓𶛐󙧽򣎈c𶚰򣎋𶚥𶚭򣎟5E_...,
personG_񬴵𶛓d񬴺󙨐񬴓󙨊𶛎𶛚...,
personO]򣎖󙨀k򣎥𶛝...,
personW[󙧸񬴭r󙨏򣎪...,
person_ZZZ]
```

These synthetic strings are then bound as prepared-statement parameters:

```
i.a.c.r.JdbcSelectQuerier$Result():79 Querying SELECT ... FROM `folkdata`.`person` WHERE (`personID` >= ?) AND (`personID` <= ?)
i.a.c.r.JdbcSelectQuerier$Result(initQueryExecution):103 Setting parameter #2 to Binding(value="person?a𶛓򣎦]𶛐...", type=StringFieldType)
```

The JDBC connection charset is utf8mb4, so MySQL 8 attempts to coerce the `utf8mb4_0900_ai_ci` parameter into the column's `latin1_swedish_ci` collation (to use the PK index) and fails with server error 3988, because the interpolated code points are not representable in latin1.

### Stack trace

```
java.sql.SQLException: Conversion from collation utf8mb4_0900_ai_ci into latin1_swedish_ci impossible for parameter
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:114)
at com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:612)
at com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:320)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1056)
at io.airbyte.cdk.read.JdbcSelectQuerier$Result.initQueryExecution(SelectQuerier.kt:107)
at io.airbyte.cdk.read.JdbcSelectQuerier$Result.(SelectQuerier.kt:81)
at io.airbyte.cdk.read.JdbcSelectQuerier.executeQuery(SelectQuerier.kt:58)
at io.airbyte.integrations.source.mysql.MySqlSourceSelectQuerier.executeQuery(MySqlSourceSelectQuerier.kt:31)
at io.airbyte.cdk.read.JdbcNonResumablePartitionReader.run(JdbcPartitionReader.kt:131)
at io.airbyte.cdk.read.FeedReader$readPartitionWithResources$4.invokeSuspend(FeedReader.kt:283)
...
```

`failureOrigin: source` on all 4 attempts; the retries can never succeed because the boundary interpolation deterministically produces non-latin1 code points.

### Conditions to reproduce

1. A table large enough to trigger the concurrent partition split (> ~3000 MiB estimated), AND
2. a string (varchar) primary key on a column whose charset is latin1 (or any charset smaller than utf8mb4), AND
3. PK values that are not GUID-shaped (so the GUID-specific handling from #69104 / #76050 does not apply — our keys look like `person`).

Setting connector concurrency to 1 does **not** avoid the bug — the log shows `Effective concurrency: 1` yet the table is still split into 6 partitions, and each partition query still uses the synthetic boundaries.

### Expected behavior

Partition boundary strings should be usable in comparisons against the PK column. Possible fixes:

- Clamp interpolated code points to the column's character set (for latin1, code points ≤ U+00FF), or
- use actual sampled key values as boundaries instead of synthetic midpoints (as source-mysql 3.11.x did), or
- wrap the parameter in `CONVERT(? USING )` / add an explicit `COLLATE` so the server can compare without coercing the parameter.

### Workaround

Downgrading the connection's source-mysql to 3.11.21 avoids the issue (its chunking uses real key values). This matches the workaround reported for the related partition-boundary regressions in #64909 / #64139.

### Related issues

- #64909 ([source-mysql] Regression in 3.50.x: ArithmeticException in partition boundary calculation)
- #64139 ([source-mysql] java.lang.ArithmeticException: / by zero)
- #69104 (Better partitioning for tables with GUID string primary key — does not cover non-GUID string PKs)

### Relevant log output

```shell
java.sql.SQLException: Conversion from collation utf8mb4_0900_ai_ci into latin1_swedish_ci impossible for parameter
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:114)
at com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:612)
at com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:320)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1056)
at io.airbyte.cdk.read.JdbcSelectQuerier$Result.initQueryExecution(SelectQuerier.kt:107)
at io.airbyte.cdk.read.JdbcSelectQuerier$Result.(SelectQuerier.kt:81)
at io.airbyte.cdk.read.JdbcSelectQuerier.executeQuery(SelectQuerier.kt:58)
at io.airbyte.integrations.source.mysql.MySqlSourceSelectQuerier.executeQuery(MySqlSourceSelectQuerier.kt:31)
at io.airbyte.cdk.read.JdbcNonResumablePartitionReader.run(JdbcPartitionReader.kt:131)
at io.airbyte.cdk.read.FeedReader$readPartitionWithResources$4.invokeSuspend(FeedReader.kt:283)
...
```

### Contribute

- [ ] Yes, I want to contribute

---
**Internal Tracking:** airbytehq/oncall#13104

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.