airbytehq / airbytehq/airbyte

[source-mysql] Schema discovery fails with `Duplicate key item_id` on Azure MySQL when all tables share the same primary key column name

Offen
#76,233 3 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
area/connectors autoteam community connectors/source/mysql needs-triage team/extensibility type/bug
Vorherrschende Sprache
Python
Sterne
22.1k
Forks
5.3k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

### Connector Name

source-mysql

### Connector Version

3.51.4

### What step the error happened?

During the sync

### Relevant information

### What step did the error happen?
Schema discovery — when adding a new MySQL source and attempting to fetch the schema.

### Relevant information

Schema discovery fails with `java.lang.IllegalStateException: Duplicate key item_id` when connecting to an Azure-hosted MySQL database. All 15 tables in the schema have `item_id` as their sole primary key column.

The schema itself is clean — there are no actual duplicate columns. Both `INFORMATION_SCHEMA.COLUMNS` and `INFORMATION_SCHEMA.STATISTICS` confirm a single `item_id` entry per table with no duplicate indexes. No views exist — all 15 objects are base tables.

The error appears to be caused by the JDBC metadata introspection combining column metadata and primary key metadata, resulting in `item_id` appearing twice in the field list passed to `CatalogHelpers.fieldsToJsonSchema()`. This blocks source creation entirely as it occurs before the stream selector is shown.

### Error log

```
java.lang.IllegalStateException: Duplicate key item_id (attempted merging values {airbyte_type=integer, type=number} and {airbyte_type=integer, type=number})
at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:135)
at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:182)
at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at io.airbyte.protocol.models.v0.CatalogHelpers.fieldsToJsonSchema(CatalogHelpers.java:205)
at io.airbyte.protocol.models.v0.CatalogHelpers.createAirbyteStream(CatalogHelpers.java:53)
at io.airbyte.cdk.discover.AirbyteStreamFactory$Companion.createAirbyteStream(AirbyteStreamFactory.kt:17)
at io.airbyte.cdk.discover.JdbcAirbyteStreamFactory.create(JdbcAirbyteStreamFactory.kt:36)
at io.airbyte.cdk.discover.DiscoverOperation.execute(DiscoverOperation.kt:41)
at io.airbyte.cdk.AirbyteConnectorRunnable.run(AirbyteConnectorRunnable.kt:36)
at picocli.CommandLine.executeUserObject(CommandLine.java:2030)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2457)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2419)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277)
```

### Schema verification

The following queries confirm the schema is clean:

```sql
-- Returns 0 rows
SELECT COLUMN_NAME, COUNT(*)
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = 'power-bi'
GROUP BY TABLE_NAME, COLUMN_NAME
HAVING COUNT(*) > 1;

-- Returns exactly one PRIMARY index per table
SELECT TABLE_NAME, INDEX_NAME, COLUMN_NAME, SEQ_IN_INDEX
FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = 'power-bi'
AND COLUMN_NAME = 'item_id'
ORDER BY TABLE_NAME;
```

All 15 tables have `item_id` at `ORDINAL_POSITION = 1`, `COLUMN_KEY = PRI`, with a single `PRIMARY` index entry each.

### Expected behaviour
Schema discovery completes successfully and all 15 tables are available for stream selection.

### Actual behaviour
Discovery fails with `IllegalStateException: Duplicate key item_id` and the source cannot be configured.

### Additional context
- Database hosted on **Azure Database for MySQL** (`mysql.database.azure.com`) — Azure's JDBC driver behaviour may be a contributing factor
- SSL mode: `required`
- 15 tables, all base tables (no views), all using `item_id` as their sole PK column

### Relevant log output

```shell
java.lang.IllegalStateException: Duplicate key item_id (attempted merging values {airbyte_type=integer, type=number} and {airbyte_type=integer, type=number})
at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:135)
at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:182)
at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at io.airbyte.protocol.models.v0.CatalogHelpers.fieldsToJsonSchema(CatalogHelpers.java:205)
at io.airbyte.protocol.models.v0.CatalogHelpers.createAirbyteStream(CatalogHelpers.java:53)
at io.airbyte.cdk.discover.AirbyteStreamFactory$Companion.createAirbyteStream(AirbyteStreamFactory.kt:17)
at io.airbyte.cdk.discover.JdbcAirbyteStreamFactory.create(JdbcAirbyteStreamFactory.kt:36)
at io.airbyte.cdk.discover.DiscoverOperation.execute(DiscoverOperation.kt:41)
at io.airbyte.cdk.AirbyteConnectorRunnable.run(AirbyteConnectorRunnable.kt:36)
at picocli.CommandLine.executeUserObject(CommandLine.java:2030)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2457)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2419)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277)
```

### Contribute

- [ ] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11947

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.