airbytehq / airbytehq/airbyte

[source-mysql] Schema discovery fails with `Duplicate key item_id` on Azure MySQL when all tables share the same primary key column name

未關閉
#76,233 3 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
area/connectors autoteam community connectors/source/mysql needs-triage team/extensibility type/bug
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

### Connector Name

source-mysql

### Connector Version

3.51.4

### What step the error happened?

During the sync

### Relevant information

### What step did the error happen?
Schema discovery — when adding a new MySQL source and attempting to fetch the schema.

### Relevant information

Schema discovery fails with `java.lang.IllegalStateException: Duplicate key item_id` when connecting to an Azure-hosted MySQL database. All 15 tables in the schema have `item_id` as their sole primary key column.

The schema itself is clean — there are no actual duplicate columns. Both `INFORMATION_SCHEMA.COLUMNS` and `INFORMATION_SCHEMA.STATISTICS` confirm a single `item_id` entry per table with no duplicate indexes. No views exist — all 15 objects are base tables.

The error appears to be caused by the JDBC metadata introspection combining column metadata and primary key metadata, resulting in `item_id` appearing twice in the field list passed to `CatalogHelpers.fieldsToJsonSchema()`. This blocks source creation entirely as it occurs before the stream selector is shown.

### Error log

```
java.lang.IllegalStateException: Duplicate key item_id (attempted merging values {airbyte_type=integer, type=number} and {airbyte_type=integer, type=number})
at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:135)
at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:182)
at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at io.airbyte.protocol.models.v0.CatalogHelpers.fieldsToJsonSchema(CatalogHelpers.java:205)
at io.airbyte.protocol.models.v0.CatalogHelpers.createAirbyteStream(CatalogHelpers.java:53)
at io.airbyte.cdk.discover.AirbyteStreamFactory$Companion.createAirbyteStream(AirbyteStreamFactory.kt:17)
at io.airbyte.cdk.discover.JdbcAirbyteStreamFactory.create(JdbcAirbyteStreamFactory.kt:36)
at io.airbyte.cdk.discover.DiscoverOperation.execute(DiscoverOperation.kt:41)
at io.airbyte.cdk.AirbyteConnectorRunnable.run(AirbyteConnectorRunnable.kt:36)
at picocli.CommandLine.executeUserObject(CommandLine.java:2030)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2457)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2419)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277)
```

### Schema verification

The following queries confirm the schema is clean:

```sql
-- Returns 0 rows
SELECT COLUMN_NAME, COUNT(*)
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = 'power-bi'
GROUP BY TABLE_NAME, COLUMN_NAME
HAVING COUNT(*) > 1;

-- Returns exactly one PRIMARY index per table
SELECT TABLE_NAME, INDEX_NAME, COLUMN_NAME, SEQ_IN_INDEX
FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = 'power-bi'
AND COLUMN_NAME = 'item_id'
ORDER BY TABLE_NAME;
```

All 15 tables have `item_id` at `ORDINAL_POSITION = 1`, `COLUMN_KEY = PRI`, with a single `PRIMARY` index entry each.

### Expected behaviour
Schema discovery completes successfully and all 15 tables are available for stream selection.

### Actual behaviour
Discovery fails with `IllegalStateException: Duplicate key item_id` and the source cannot be configured.

### Additional context
- Database hosted on **Azure Database for MySQL** (`mysql.database.azure.com`) — Azure's JDBC driver behaviour may be a contributing factor
- SSL mode: `required`
- 15 tables, all base tables (no views), all using `item_id` as their sole PK column

### Relevant log output

```shell
java.lang.IllegalStateException: Duplicate key item_id (attempted merging values {airbyte_type=integer, type=number} and {airbyte_type=integer, type=number})
at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:135)
at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:182)
at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at io.airbyte.protocol.models.v0.CatalogHelpers.fieldsToJsonSchema(CatalogHelpers.java:205)
at io.airbyte.protocol.models.v0.CatalogHelpers.createAirbyteStream(CatalogHelpers.java:53)
at io.airbyte.cdk.discover.AirbyteStreamFactory$Companion.createAirbyteStream(AirbyteStreamFactory.kt:17)
at io.airbyte.cdk.discover.JdbcAirbyteStreamFactory.create(JdbcAirbyteStreamFactory.kt:36)
at io.airbyte.cdk.discover.DiscoverOperation.execute(DiscoverOperation.kt:41)
at io.airbyte.cdk.AirbyteConnectorRunnable.run(AirbyteConnectorRunnable.kt:36)
at picocli.CommandLine.executeUserObject(CommandLine.java:2030)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2457)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2419)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277)
```

### Contribute

- [ ] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11947

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。