[source-mysql] Concurrent initial-load partitioning generates non-latin1 boundary strings for varchar PK on latin1 table → error 3988 "Conversion from collation utf8mb4_0900_ai_ci into latin1_swedish_ci impossible for parameter"
- Ngôn ngữ chính
- Python
- Star
- 22.1k
- Fork
- 5.3k
- Chỉ số merge pull request
- Chỉ số pull request đang chờ
Mô tả
### Connector Name
source-mysql
### Connector Version
3.53.1
### What step the error happened?
During the sync
## Environment
- Airbyte OSS 1.6.0 (Helm deployment on EKS)
- source-mysql 3.53.1
- destination-snowflake 4.0.4
- MySQL 8.0.42 (Aurora RDS)
- Sync mode: Incremental | Append + Deduped, CDC (read changes using binary log)
## Relevant information
The initial snapshot of one specific table fails on every attempt (4/4) with:
```
java.sql.SQLException: Conversion from collation utf8mb4_0900_ai_ci into latin1_swedish_ci impossible for parameter
```
The failing table is large (~13.6 GB estimated) and has a **`varchar(128)` primary key on a table with `DEFAULT CHARSET=latin1`** (PK collation `latin1_swedish_ci`):
```sql
CREATE TABLE `person` (
`personID` varchar(128) NOT NULL,
-- ... ~45 more columns ...
PRIMARY KEY (`personID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
```
### Root cause analysis
Because the table exceeds the 3000 MiB target partition size, `MySqlJdbcConcurrentPartitionsCreator` splits the snapshot into 6 partitions:
```
i.a.i.s.m.MySqlJdbcConcurrentPartitionsCreator(run):521 Table memory size estimated at 13632 MiB.
i.a.i.s.m.MySqlJdbcConcurrentPartitionsCreator(run):522 Target partition size is 3000 MiB.
i.a.i.s.m.MySqlJdbcConcurrentPartitionsCreator(run):551 Table will be read by 6 concurrent partition reader(s).
```
Since the PK is a string, `MySqlSourceJdbcPartitionFactory.internalCalculateBoundaries` synthesizes midpoint boundary strings by interpolating between the sampled min/max keys. The real bounds are plain ASCII, but the four computed midpoints contain arbitrary Unicode code points, including supplementary-plane characters:
```
i.a.i.s.m.MySqlSourceJdbcPartitionFactory(internalCalculateBoundaries):702 boundaries:
[person7cry,
person?a]c5E_...,
personG_d...,
personO]k...,
personW[r...,
person_ZZZ]
```
These synthetic strings are then bound as prepared-statement parameters:
```
i.a.c.r.JdbcSelectQuerier$Result():79 Querying SELECT ... FROM `folkdata`.`person` WHERE (`personID` >= ?) AND (`personID` <= ?)
i.a.c.r.JdbcSelectQuerier$Result(initQueryExecution):103 Setting parameter #2 to Binding(value="person?a]...", type=StringFieldType)
```
The JDBC connection charset is utf8mb4, so MySQL 8 attempts to coerce the `utf8mb4_0900_ai_ci` parameter into the column's `latin1_swedish_ci` collation (to use the PK index) and fails with server error 3988, because the interpolated code points are not representable in latin1.
### Stack trace
```
java.sql.SQLException: Conversion from collation utf8mb4_0900_ai_ci into latin1_swedish_ci impossible for parameter
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:114)
at com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:612)
at com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:320)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1056)
at io.airbyte.cdk.read.JdbcSelectQuerier$Result.initQueryExecution(SelectQuerier.kt:107)
at io.airbyte.cdk.read.JdbcSelectQuerier$Result.(SelectQuerier.kt:81)
at io.airbyte.cdk.read.JdbcSelectQuerier.executeQuery(SelectQuerier.kt:58)
at io.airbyte.integrations.source.mysql.MySqlSourceSelectQuerier.executeQuery(MySqlSourceSelectQuerier.kt:31)
at io.airbyte.cdk.read.JdbcNonResumablePartitionReader.run(JdbcPartitionReader.kt:131)
at io.airbyte.cdk.read.FeedReader$readPartitionWithResources$4.invokeSuspend(FeedReader.kt:283)
...
```
`failureOrigin: source` on all 4 attempts; the retries can never succeed because the boundary interpolation deterministically produces non-latin1 code points.
### Conditions to reproduce
1. A table large enough to trigger the concurrent partition split (> ~3000 MiB estimated), AND
2. a string (varchar) primary key on a column whose charset is latin1 (or any charset smaller than utf8mb4), AND
3. PK values that are not GUID-shaped (so the GUID-specific handling from #69104 / #76050 does not apply — our keys look like `person`).
Setting connector concurrency to 1 does **not** avoid the bug — the log shows `Effective concurrency: 1` yet the table is still split into 6 partitions, and each partition query still uses the synthetic boundaries.
### Expected behavior
Partition boundary strings should be usable in comparisons against the PK column. Possible fixes:
- Clamp interpolated code points to the column's character set (for latin1, code points ≤ U+00FF), or
- use actual sampled key values as boundaries instead of synthetic midpoints (as source-mysql 3.11.x did), or
- wrap the parameter in `CONVERT(? USING )` / add an explicit `COLLATE` so the server can compare without coercing the parameter.
### Workaround
Downgrading the connection's source-mysql to 3.11.21 avoids the issue (its chunking uses real key values). This matches the workaround reported for the related partition-boundary regressions in #64909 / #64139.
### Related issues
- #64909 ([source-mysql] Regression in 3.50.x: ArithmeticException in partition boundary calculation)
- #64139 ([source-mysql] java.lang.ArithmeticException: / by zero)
- #69104 (Better partitioning for tables with GUID string primary key — does not cover non-GUID string PKs)
### Relevant log output
```shell
java.sql.SQLException: Conversion from collation utf8mb4_0900_ai_ci into latin1_swedish_ci impossible for parameter
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:114)
at com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:612)
at com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:320)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1056)
at io.airbyte.cdk.read.JdbcSelectQuerier$Result.initQueryExecution(SelectQuerier.kt:107)
at io.airbyte.cdk.read.JdbcSelectQuerier$Result.(SelectQuerier.kt:81)
at io.airbyte.cdk.read.JdbcSelectQuerier.executeQuery(SelectQuerier.kt:58)
at io.airbyte.integrations.source.mysql.MySqlSourceSelectQuerier.executeQuery(MySqlSourceSelectQuerier.kt:31)
at io.airbyte.cdk.read.JdbcNonResumablePartitionReader.run(JdbcPartitionReader.kt:131)
at io.airbyte.cdk.read.FeedReader$readPartitionWithResources$4.invokeSuspend(FeedReader.kt:283)
...
```
### Contribute
- [ ] Yes, I want to contribute
---
**Internal Tracking:** airbytehq/oncall#13104
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.