destination-bigquery: STRUCT vs JSON type mismatch on customer_journey_summary column during dedup merge
- Lingua principale
- Python
- Stelle
- 22.1k
- Fork
- 5.3k
- Metriche di merge delle PR
- Metriche PR in attesa
Descrizione
### Connector Name
destination-bigquery
### Connector Version
3.0.16
### What step the error happened?
During the sync
### Relevant information
When syncing the customer_journey_summary stream from Shopify source in Dedupe mode, the sync fails during the final merge step with:
```
BigQueryException: Query column 7 has type STRUCT<_airbyte_raw_id STRING, _airbyte_extracted_at TIMESTAMP, _airbyte_meta JSON, ...> which cannot be inserted into column customer_journey_summary, which has type JSON at [3:1]
The issue occurs because:
```
The table customer_journey_summary has a column also named customer_journey_summary containing nested object data
Airbyte creates this column as JSON type
During the dedup merge step, the connector tries to insert STRUCT type data into the JSON column
BigQuery rejects the implicit STRUCT → JSON conversion
Data loads successfully into the staging table (airbyte_internal), but fails on CloseStreamTask when merging to the final table.
Dropping and recreating tables does not fix the issue. The only workaround is switching to Append mode, which is not ideal for incremental syncs.
### Relevant log output
```shell
2025-12-29 15:48:45 replication-orchestrator INFO Sync summary: {
"status" : "failed",
"recordsSynced" : 385331,
"bytesSynced" : 605943476,
"startTime" : 1767016588236,
"endTime" : 1767019725276,
"totalStats" : {
"additionalStats" : { },
"bytesCommitted" : 605943476,
"bytesEmitted" : 797071774,
"destinationStateMessagesEmitted" : 1566,
"destinationWriteEndTime" : 0,
"destinationWriteStartTime" : 1767016593224,
"meanSecondsBeforeSourceStateMessageEmitted" : 26,
"maxSecondsBeforeSourceStateMessageEmitted" : 651,
"maxSecondsBetweenStateMessageEmittedandCommitted" : 365,
"meanSecondsBetweenStateMessageEmittedandCommitted" : 184,
"recordsEmitted" : 436394,
"recordsCommitted" : 385331,
"recordsFilteredOut" : 0,
"bytesFilteredOut" : 0,
"replicationEndTime" : 1767019725274,
"replicationStartTime" : 1767016588236,
"sourceReadEndTime" : 1767019698889,
"sourceReadStartTime" : 1767016593224,
"sourceStateMessagesEmitted" : 1771
},
"streamStats" : [ {
"streamName" : "orders",
"stats" : {
"additionalStats" : {
"sourceFieldsPopulated" : 1849762
},
"bytesCommitted" : 348990921,
"bytesEmitted" : 540119219,
"estimatedBytes" : 0,
"estimatedRecords" : 0,
"recordsEmitted" : 156813,
"recordsCommitted" : 105750,
"recordsFilteredOut" : 0,
"bytesFilteredOut" : 0
}
}, {
"streamName" : "customer_journey_summary",
"stats" : {
"additionalStats" : {
"sourceFieldsPopulated" : 470244
},
"bytesCommitted" : 232127518,
"bytesEmitted" : 232127518,
"estimatedBytes" : 0,
"estimatedRecords" : 0,
"recordsEmitted" : 156748,
"recordsCommitted" : 156748,
"recordsFilteredOut" : 0,
"bytesFilteredOut" : 0
}
}, {
"streamName" : "customers",
"stats" : {
"additionalStats" : {
"sourceFieldsPopulated" : 982198
},
"bytesCommitted" : 24825037,
"bytesEmitted" : 24825037,
"estimatedBytes" : 0,
"estimatedRecords" : 0,
"recordsEmitted" : 122833,
"recordsCommitted" : 122833,
"recordsFilteredOut" : 0,
"bytesFilteredOut" : 0
}
} ],
"performanceMetrics" : { },
"streamCount" : 3
}
2025-12-29 15:48:45 replication-orchestrator INFO Failures: [ {
"failureOrigin" : "destination",
"failureType" : "system_error",
"internalMessage" : "com.google.cloud.bigquery.BigQueryException: Query column 7 has type STRUCT<_airbyte_raw_id STRING, _airbyte_extracted_at TIMESTAMP, _airbyte_meta JSON, ...> which cannot be inserted into column customer_journey_summary, which has type JSON at [3:1]",
"externalMessage" : "Query column 7 has type STRUCT<_airbyte_raw_id STRING, _airbyte_extracted_at TIMESTAMP, _airbyte_meta JSON, ...> which cannot be inserted into column customer_journey_summary, which has type JSON at [3:1]",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 63686929,
"from_trace_message" : true,
"connector_command" : "write"
},
"stacktrace" : "com.google.cloud.bigquery.BigQueryException: Query column 7 has type STRUCT<_airbyte_raw_id STRING, _airbyte_extracted_at TIMESTAMP, _airbyte_meta JSON, ...> which cannot be inserted into column customer_journey_summary, which has type JSON at [3:1]\n\tat com.google.cloud.bigquery.Job.reload(Job.java:591)\n\tat io.airbyte.integrations.destination.bigquery.write.typing_deduping.BigQueryDatabaseHandler.execute(BigQueryDatabaseHandler.kt:102)\n\tat io.airbyte.integrations.destination.bigquery.write.typing_deduping.direct_load_tables.BigqueryTableSchemaEvolutionClient.recreateTable-072v0zg(BigqueryTableSchemaEvolutionClient.kt:289)\n\tat io.airbyte.integrations.destination.bigquery.write.typing_deduping.direct_load_tables.BigqueryTableSchemaEvolutionClient.ensureSchemaMatches-Uhgc1Ac(BigqueryTableSchemaEvolutionClient.kt:65)\n\tat io.airbyte.cdk.load.orchestration.db.direct_load_table.DirectLoadTableDedupStreamLoader.close(DirectLoadTableStreamLoader.kt:112)\n\tat io.airbyte.cdk.load.write.StreamLoader.close$default(StreamLoader.kt:28)\n\tat io.airbyte.cdk.load.task.implementor.CloseStreamTask.execute(CloseStreamTask.kt:24)\n\tat io.airbyte.cdk.load.task.DestinationTaskLauncher$WrappedTask.execute(DestinationTaskLauncher.kt:124)\n\tat io.airbyte.cdk.load.task.TaskScopeProvider$launch$job$1.invokeSuspend(TaskScopeProvider.kt:35)\n\tat kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)\n\tat kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)\n\tat kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)\n\tat kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)\n",
"timestamp" : 1767019713437
}, {
"failureOrigin" : "destination",
"internalMessage" : "Destination process exited with non-zero exit code 1",
"externalMessage" : "Something went wrong within the destination connector",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 63686929,
"connector_command" : "write"
},
"stacktrace" : "io.airbyte.workers.internal.exception.DestinationException: Destination process exited with non-zero exit code 1\n\tat io.airbyte.container.orchestrator.worker.DestinationReader.run(ReplicationTask.kt:55)\n\tat io.airbyte.container.orchestrator.worker.ReplicationWorker$runJobs$2$tasks$1$1.invokeSuspend(ReplicationWorker.kt:152)\n\tat io.airbyte.container.orchestrator.worker.ReplicationWorker$runJobs$2$tasks$1$1.invoke(ReplicationWorker.kt)\n\tat io.airbyte.container.orchestrator.worker.ReplicationWorker$runJobs$2$tasks$1$1.invoke(ReplicationWorker.kt)\n\tat io.airbyte.container.orchestrator.worker.util.AsyncUtils$runAsync$1$1.invokeSuspend(AsyncUtils.kt:22)\n\tat io.airbyte.container.orchestrator.worker.util.AsyncUtils$runAsync$1$1.invoke(AsyncUtils.kt)\n\tat io.airbyte.container.orchestrator.worker.util.AsyncUtils$runAsync$1$1.invoke(AsyncUtils.kt)\n\tat kotlinx.coroutines.intrinsics.UndispatchedKt.startUndspatched(Undispatched.kt:66)\n\tat kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:43)\n\tat kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:165)\n\tat kotlinx.coroutines.BuildersKt.withContext(Unknown Source)\n\tat io.airbyte.container.orchestrator.worker.util.AsyncUtils$runAsync$1.invokeSuspend(AsyncUtils.kt:21)\n\tat kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)\n\tat kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)\n\tat io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)\n\tat io.micrometer.core.instrument.Timer.lambda$wrap$2(Timer.java:199)\n\tat datadog.trace.bootstrap.instrumentation.java.concurrent.Wrapper.run(Wrapper.java:47)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n",
"timestamp" : 1767019725274
} ]
2025-12-29 15:48:45 replication-orchestrator INFO
2025-12-29 15:48:45 replication-orchestrator INFO Closing io.micronaut.context.propagation.instrument.execution.ContextPropagatingExecutorService@177b38b9
2025-12-29 15:48:45 replication-orchestrator INFO ----- END REPLICATION -----
2025-12-29 15:48:45 replication-orchestrator INFO
```
### Contribute
- [x] Yes, I want to contribute
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.