[source-mysql] Schema discovery fails with `Duplicate key item_id` on Azure MySQL when all tables share the same primary key column name
- Vorherrschende Sprache
- Python
- Sterne
- 22.1k
- Forks
- 5.3k
- PR-Merge-Kennzahlen
- PR-Kennzahlen ausstehend
Beschreibung
### Connector Name
source-mysql
### Connector Version
3.51.4
### What step the error happened?
During the sync
### Relevant information
### What step did the error happen?
Schema discovery — when adding a new MySQL source and attempting to fetch the schema.
### Relevant information
Schema discovery fails with `java.lang.IllegalStateException: Duplicate key item_id` when connecting to an Azure-hosted MySQL database. All 15 tables in the schema have `item_id` as their sole primary key column.
The schema itself is clean — there are no actual duplicate columns. Both `INFORMATION_SCHEMA.COLUMNS` and `INFORMATION_SCHEMA.STATISTICS` confirm a single `item_id` entry per table with no duplicate indexes. No views exist — all 15 objects are base tables.
The error appears to be caused by the JDBC metadata introspection combining column metadata and primary key metadata, resulting in `item_id` appearing twice in the field list passed to `CatalogHelpers.fieldsToJsonSchema()`. This blocks source creation entirely as it occurs before the stream selector is shown.
### Error log
```
java.lang.IllegalStateException: Duplicate key item_id (attempted merging values {airbyte_type=integer, type=number} and {airbyte_type=integer, type=number})
at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:135)
at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:182)
at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at io.airbyte.protocol.models.v0.CatalogHelpers.fieldsToJsonSchema(CatalogHelpers.java:205)
at io.airbyte.protocol.models.v0.CatalogHelpers.createAirbyteStream(CatalogHelpers.java:53)
at io.airbyte.cdk.discover.AirbyteStreamFactory$Companion.createAirbyteStream(AirbyteStreamFactory.kt:17)
at io.airbyte.cdk.discover.JdbcAirbyteStreamFactory.create(JdbcAirbyteStreamFactory.kt:36)
at io.airbyte.cdk.discover.DiscoverOperation.execute(DiscoverOperation.kt:41)
at io.airbyte.cdk.AirbyteConnectorRunnable.run(AirbyteConnectorRunnable.kt:36)
at picocli.CommandLine.executeUserObject(CommandLine.java:2030)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2457)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2419)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277)
```
### Schema verification
The following queries confirm the schema is clean:
```sql
-- Returns 0 rows
SELECT COLUMN_NAME, COUNT(*)
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = 'power-bi'
GROUP BY TABLE_NAME, COLUMN_NAME
HAVING COUNT(*) > 1;
-- Returns exactly one PRIMARY index per table
SELECT TABLE_NAME, INDEX_NAME, COLUMN_NAME, SEQ_IN_INDEX
FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = 'power-bi'
AND COLUMN_NAME = 'item_id'
ORDER BY TABLE_NAME;
```
All 15 tables have `item_id` at `ORDINAL_POSITION = 1`, `COLUMN_KEY = PRI`, with a single `PRIMARY` index entry each.
### Expected behaviour
Schema discovery completes successfully and all 15 tables are available for stream selection.
### Actual behaviour
Discovery fails with `IllegalStateException: Duplicate key item_id` and the source cannot be configured.
### Additional context
- Database hosted on **Azure Database for MySQL** (`mysql.database.azure.com`) — Azure's JDBC driver behaviour may be a contributing factor
- SSL mode: `required`
- 15 tables, all base tables (no views), all using `item_id` as their sole PK column
### Relevant log output
```shell
java.lang.IllegalStateException: Duplicate key item_id (attempted merging values {airbyte_type=integer, type=number} and {airbyte_type=integer, type=number})
at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:135)
at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:182)
at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at io.airbyte.protocol.models.v0.CatalogHelpers.fieldsToJsonSchema(CatalogHelpers.java:205)
at io.airbyte.protocol.models.v0.CatalogHelpers.createAirbyteStream(CatalogHelpers.java:53)
at io.airbyte.cdk.discover.AirbyteStreamFactory$Companion.createAirbyteStream(AirbyteStreamFactory.kt:17)
at io.airbyte.cdk.discover.JdbcAirbyteStreamFactory.create(JdbcAirbyteStreamFactory.kt:36)
at io.airbyte.cdk.discover.DiscoverOperation.execute(DiscoverOperation.kt:41)
at io.airbyte.cdk.AirbyteConnectorRunnable.run(AirbyteConnectorRunnable.kt:36)
at picocli.CommandLine.executeUserObject(CommandLine.java:2030)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2457)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2419)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277)
```
### Contribute
- [ ] Yes, I want to contribute
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11947
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.