airbytehq / airbytehq/airbyte

destination-bigquery: STRUCT vs JSON type mismatch on customer_journey_summary column during dedup merge

Aperta
#71,043 2 commenti 1 reazione 0 assegnatari Vedi su GitHub
area/connectors autoteam community connectors/destination/bigquery needs-triage team/extensibility type/bug
Lingua principale
Python
Stelle
22.1k
Fork
5.3k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

### Connector Name

destination-bigquery

### Connector Version

3.0.16

### What step the error happened?

During the sync

### Relevant information

When syncing the customer_journey_summary stream from Shopify source in Dedupe mode, the sync fails during the final merge step with:

```
BigQueryException: Query column 7 has type STRUCT<_airbyte_raw_id STRING, _airbyte_extracted_at TIMESTAMP, _airbyte_meta JSON, ...> which cannot be inserted into column customer_journey_summary, which has type JSON at [3:1]
The issue occurs because:
```

The table customer_journey_summary has a column also named customer_journey_summary containing nested object data
Airbyte creates this column as JSON type
During the dedup merge step, the connector tries to insert STRUCT type data into the JSON column
BigQuery rejects the implicit STRUCT → JSON conversion

Data loads successfully into the staging table (airbyte_internal), but fails on CloseStreamTask when merging to the final table.
Dropping and recreating tables does not fix the issue. The only workaround is switching to Append mode, which is not ideal for incremental syncs.

### Relevant log output

```shell
2025-12-29 15:48:45 replication-orchestrator INFO Sync summary: {
"status" : "failed",
"recordsSynced" : 385331,
"bytesSynced" : 605943476,
"startTime" : 1767016588236,
"endTime" : 1767019725276,
"totalStats" : {
"additionalStats" : { },
"bytesCommitted" : 605943476,
"bytesEmitted" : 797071774,
"destinationStateMessagesEmitted" : 1566,
"destinationWriteEndTime" : 0,
"destinationWriteStartTime" : 1767016593224,
"meanSecondsBeforeSourceStateMessageEmitted" : 26,
"maxSecondsBeforeSourceStateMessageEmitted" : 651,
"maxSecondsBetweenStateMessageEmittedandCommitted" : 365,
"meanSecondsBetweenStateMessageEmittedandCommitted" : 184,
"recordsEmitted" : 436394,
"recordsCommitted" : 385331,
"recordsFilteredOut" : 0,
"bytesFilteredOut" : 0,
"replicationEndTime" : 1767019725274,
"replicationStartTime" : 1767016588236,
"sourceReadEndTime" : 1767019698889,
"sourceReadStartTime" : 1767016593224,
"sourceStateMessagesEmitted" : 1771
},
"streamStats" : [ {
"streamName" : "orders",
"stats" : {
"additionalStats" : {
"sourceFieldsPopulated" : 1849762
},
"bytesCommitted" : 348990921,
"bytesEmitted" : 540119219,
"estimatedBytes" : 0,
"estimatedRecords" : 0,
"recordsEmitted" : 156813,
"recordsCommitted" : 105750,
"recordsFilteredOut" : 0,
"bytesFilteredOut" : 0
}
}, {
"streamName" : "customer_journey_summary",
"stats" : {
"additionalStats" : {
"sourceFieldsPopulated" : 470244
},
"bytesCommitted" : 232127518,
"bytesEmitted" : 232127518,
"estimatedBytes" : 0,
"estimatedRecords" : 0,
"recordsEmitted" : 156748,
"recordsCommitted" : 156748,
"recordsFilteredOut" : 0,
"bytesFilteredOut" : 0
}
}, {
"streamName" : "customers",
"stats" : {
"additionalStats" : {
"sourceFieldsPopulated" : 982198
},
"bytesCommitted" : 24825037,
"bytesEmitted" : 24825037,
"estimatedBytes" : 0,
"estimatedRecords" : 0,
"recordsEmitted" : 122833,
"recordsCommitted" : 122833,
"recordsFilteredOut" : 0,
"bytesFilteredOut" : 0
}
} ],
"performanceMetrics" : { },
"streamCount" : 3
}
2025-12-29 15:48:45 replication-orchestrator INFO Failures: [ {
"failureOrigin" : "destination",
"failureType" : "system_error",
"internalMessage" : "com.google.cloud.bigquery.BigQueryException: Query column 7 has type STRUCT<_airbyte_raw_id STRING, _airbyte_extracted_at TIMESTAMP, _airbyte_meta JSON, ...> which cannot be inserted into column customer_journey_summary, which has type JSON at [3:1]",
"externalMessage" : "Query column 7 has type STRUCT<_airbyte_raw_id STRING, _airbyte_extracted_at TIMESTAMP, _airbyte_meta JSON, ...> which cannot be inserted into column customer_journey_summary, which has type JSON at [3:1]",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 63686929,
"from_trace_message" : true,
"connector_command" : "write"
},
"stacktrace" : "com.google.cloud.bigquery.BigQueryException: Query column 7 has type STRUCT<_airbyte_raw_id STRING, _airbyte_extracted_at TIMESTAMP, _airbyte_meta JSON, ...> which cannot be inserted into column customer_journey_summary, which has type JSON at [3:1]\n\tat com.google.cloud.bigquery.Job.reload(Job.java:591)\n\tat io.airbyte.integrations.destination.bigquery.write.typing_deduping.BigQueryDatabaseHandler.execute(BigQueryDatabaseHandler.kt:102)\n\tat io.airbyte.integrations.destination.bigquery.write.typing_deduping.direct_load_tables.BigqueryTableSchemaEvolutionClient.recreateTable-072v0zg(BigqueryTableSchemaEvolutionClient.kt:289)\n\tat io.airbyte.integrations.destination.bigquery.write.typing_deduping.direct_load_tables.BigqueryTableSchemaEvolutionClient.ensureSchemaMatches-Uhgc1Ac(BigqueryTableSchemaEvolutionClient.kt:65)\n\tat io.airbyte.cdk.load.orchestration.db.direct_load_table.DirectLoadTableDedupStreamLoader.close(DirectLoadTableStreamLoader.kt:112)\n\tat io.airbyte.cdk.load.write.StreamLoader.close$default(StreamLoader.kt:28)\n\tat io.airbyte.cdk.load.task.implementor.CloseStreamTask.execute(CloseStreamTask.kt:24)\n\tat io.airbyte.cdk.load.task.DestinationTaskLauncher$WrappedTask.execute(DestinationTaskLauncher.kt:124)\n\tat io.airbyte.cdk.load.task.TaskScopeProvider$launch$job$1.invokeSuspend(TaskScopeProvider.kt:35)\n\tat kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)\n\tat kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)\n\tat kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)\n\tat kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)\n",
"timestamp" : 1767019713437
}, {
"failureOrigin" : "destination",
"internalMessage" : "Destination process exited with non-zero exit code 1",
"externalMessage" : "Something went wrong within the destination connector",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 63686929,
"connector_command" : "write"
},
"stacktrace" : "io.airbyte.workers.internal.exception.DestinationException: Destination process exited with non-zero exit code 1\n\tat io.airbyte.container.orchestrator.worker.DestinationReader.run(ReplicationTask.kt:55)\n\tat io.airbyte.container.orchestrator.worker.ReplicationWorker$runJobs$2$tasks$1$1.invokeSuspend(ReplicationWorker.kt:152)\n\tat io.airbyte.container.orchestrator.worker.ReplicationWorker$runJobs$2$tasks$1$1.invoke(ReplicationWorker.kt)\n\tat io.airbyte.container.orchestrator.worker.ReplicationWorker$runJobs$2$tasks$1$1.invoke(ReplicationWorker.kt)\n\tat io.airbyte.container.orchestrator.worker.util.AsyncUtils$runAsync$1$1.invokeSuspend(AsyncUtils.kt:22)\n\tat io.airbyte.container.orchestrator.worker.util.AsyncUtils$runAsync$1$1.invoke(AsyncUtils.kt)\n\tat io.airbyte.container.orchestrator.worker.util.AsyncUtils$runAsync$1$1.invoke(AsyncUtils.kt)\n\tat kotlinx.coroutines.intrinsics.UndispatchedKt.startUndspatched(Undispatched.kt:66)\n\tat kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:43)\n\tat kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:165)\n\tat kotlinx.coroutines.BuildersKt.withContext(Unknown Source)\n\tat io.airbyte.container.orchestrator.worker.util.AsyncUtils$runAsync$1.invokeSuspend(AsyncUtils.kt:21)\n\tat kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)\n\tat kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)\n\tat io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)\n\tat io.micrometer.core.instrument.Timer.lambda$wrap$2(Timer.java:199)\n\tat datadog.trace.bootstrap.instrumentation.java.concurrent.Wrapper.run(Wrapper.java:47)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n",
"timestamp" : 1767019725274
} ]
2025-12-29 15:48:45 replication-orchestrator INFO
2025-12-29 15:48:45 replication-orchestrator INFO Closing io.micronaut.context.propagation.instrument.execution.ContextPropagatingExecutorService@177b38b9
2025-12-29 15:48:45 replication-orchestrator INFO ----- END REPLICATION -----
2025-12-29 15:48:45 replication-orchestrator INFO
```

### Contribute

- [x] Yes, I want to contribute

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.