airbytehq / airbytehq/airbyte

[destination-bigquery][source-jira] Sync hangs forever, saying nothing to do. When cancel sync, it says one message was there

Abierto
#58,082 6 comentarios 1 reacción 0 asignados Ver en GitHub
area/connectors community team/destinations type/bug
Lenguaje dominante
Python
Estrellas
22.1k
Forks
5.3k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

### Connector Name

destination-bigquery

### Connector Version

2.10.0

### Airbyte 1.4.1

### What step the error happened?

During the sync

### Relevant information

I have a sync (jira->bigquery) that hangs very often. I can see when it hangs it prints "Batch contains: 0 records, 0 bytes bytes" and "Received null value for Job, nothing to waitFor" continuously forever. After cancelling the sync it shows: "remaining records: 1" and "Batch contains: 1 records, 45.61 KB bytes" . As it somehow it never picked up that record until I cancelled the sync

### Relevant log output

```shell
2025-04-15 10:56:13 destination WARN pool-4-thread-3 i.a.i.d.b.BigQueryUtils(waitForJobFinish):292 Received null value for Job, nothing to waitFor
2025-04-15 10:56:13 destination INFO pool-4-thread-3 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states
2025-04-15 10:56:13 destination INFO pool-4-thread-3 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete
2025-04-15 10:56:13 destination INFO pool-4-thread-3 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2025-04-15 10:56:13 destination INFO pool-4-thread-3 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2025-04-15 10:56:13 destination INFO pool-4-thread-3 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (b7fb7) -- Worker finished flushing. Current queue size: 0
2025-04-15 10:56:54 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 3 GB, allocated: 9.93 MB (9.926865577697754 MB), %% used: 0.0032314015552401543 | Queue `jira_issues_onlyissues`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 9 MB, Used: -76687 bytes, percentage Used -0.007367322719323805
2025-04-15 10:56:54 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-04-15 10:57:54 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 3 GB, allocated: 9.93 MB (9.926865577697754 MB), %% used: 0.0032314015552401543 | Queue `jira_issues_onlyissues`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 9 MB, Used: -76687 bytes, percentage Used -0.007367322719323805
2025-04-15 10:57:54 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-04-15 10:58:54 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 3 GB, allocated: 9.93 MB (9.926865577697754 MB), %% used: 0.0032314015552401543 | Queue `jira_issues_onlyissues`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 9 MB, Used: -76687 bytes, percentage Used -0.007367322719323805
2025-04-15 10:58:54 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-04-15 10:59:54 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 3 GB, allocated: 9.93 MB (9.926865577697754 MB), %% used: 0.0032314015552401543 | Queue `jira_issues_onlyissues`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 9 MB, Used: -76687 bytes, percentage Used -0.007367322719323805
2025-04-15 10:59:54 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-04-15 11:00:54 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 3 GB, allocated: 9.93 MB (9.926865577697754 MB), %% used: 0.0032314015552401543 | Queue `jira_issues_onlyissues`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 9 MB, Used: -76687 bytes, percentage Used -0.007367322719323805
2025-04-15 11:00:54 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-04-15 11:01:12 destination INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):109 flushing: trigger info: jira_airbyte - jira_issues_onlyissues, time trigger: true , size trigger: false current threshold b: 1.2 GB, queue size b: 0 bytes, penalty b: 0 bytes, after penalty b: 0 bytes
2025-04-15 11:01:12 destination INFO pool-4-thread-4 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):135 Flush Worker (dc04f) -- Worker picked up work.
2025-04-15 11:01:12 destination INFO pool-4-thread-4 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):141 Flush Worker (dc04f) -- Attempting to read from queue namespace: jira_airbyte, stream: jira_issues_onlyissues.
2025-04-15 11:01:12 destination INFO pool-4-thread-4 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2025-04-15 11:01:12 destination INFO pool-4-thread-4 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):159 Flush Worker (dc04f) -- Batch contains: 0 records, 0 bytes bytes.
2025-04-15 11:01:12 destination INFO pool-4-thread-4 i.a.i.d.b.o.BigQueryDirectLoadingStorageOperation(writeToStage):66 Writing data to table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte_internal, tableId=jira_airbyte_raw__stream_jira_issues_onlyissues_airbyte_tmp}} with schema Schema{fields=[Field{name=_airbyte_raw_id, type=STRING, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_extracted_at, type=TIMESTAMP, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_loaded_at, type=TIMESTAMP, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_data, type=STRING, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_meta, type=STRING, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_generation_id, type=INTEGER, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}]}
2025-04-15 11:01:13 destination INFO pool-4-thread-4 i.a.i.d.b.o.BigQueryDirectLoadingStorageOperation(writeToStage):79 Writing to channel completed for GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte_internal, tableId=jira_airbyte_raw__stream_jira_issues_onlyissues_airbyte_tmp}}
2025-04-15 11:01:13 destination WARN pool-4-thread-4 i.a.i.d.b.BigQueryUtils(waitForJobFinish):292 Received null value for Job, nothing to waitFor
2025-04-15 11:01:13 destination INFO pool-4-thread-4 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states
2025-04-15 11:01:13 destination INFO pool-4-thread-4 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete
2025-04-15 11:01:13 destination INFO pool-4-thread-4 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2025-04-15 11:01:13 destination INFO pool-4-thread-4 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2025-04-15 11:01:13 destination INFO pool-4-thread-4 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (dc04f) -- Worker finished flushing. Current queue size: 0
2025-04-15 11:01:54 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 3 GB, allocated: 9.93 MB (9.926865577697754 MB), %% used: 0.0032314015552401543 | Queue `jira_issues_onlyissues`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 9 MB, Used: -76687 bytes, percentage Used -0.007367322719323805
2025-04-15 11:01:54 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-04-15 11:02:54 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 3 GB, allocated: 9.93 MB (9.926865577697754 MB), %% used: 0.0032314015552401543 | Queue `jira_issues_onlyissues`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 9 MB, Used: -76687 bytes, percentage Used -0.007367322719323805
2025-04-15 11:02:54 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-04-15 11:03:54 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 3 GB, allocated: 9.93 MB (9.926865577697754 MB), %% used: 0.0032314015552401543 | Queue `jira_issues_onlyissues`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 9 MB, Used: -76687 bytes, percentage Used -0.007367322719323805
2025-04-15 11:03:54 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-04-15 11:04:54 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 3 GB, allocated: 9.93 MB (9.926865577697754 MB), %% used: 0.0032314015552401543 | Queue `jira_issues_onlyissues`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 9 MB, Used: -76687 bytes, percentage Used -0.007367322719323805
2025-04-15 11:04:54 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-04-15 11:05:54 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 3 GB, allocated: 9.93 MB (9.926865577697754 MB), %% used: 0.0032314015552401543 | Queue `jira_issues_onlyissues`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 9 MB, Used: -76687 bytes, percentage Used -0.007367322719323805
2025-04-15 11:05:54 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-04-15 11:06:13 destination INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):109 flushing: trigger info: jira_airbyte - jira_issues_onlyissues, time trigger: true , size trigger: false current threshold b: 1.2 GB, queue size b: 0 bytes, penalty b: 0 bytes, after penalty b: 0 bytes
2025-04-15 11:06:13 destination INFO pool-4-thread-5 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):135 Flush Worker (b09a1) -- Worker picked up work.
2025-04-15 11:06:13 destination INFO pool-4-thread-5 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):141 Flush Worker (b09a1) -- Attempting to read from queue namespace: jira_airbyte, stream: jira_issues_onlyissues.
2025-04-15 11:06:13 destination INFO pool-4-thread-5 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2025-04-15 11:06:13 destination INFO pool-4-thread-5 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):159 Flush Worker (b09a1) -- Batch contains: 0 records, 0 bytes bytes.
2025-04-15 11:06:13 destination INFO pool-4-thread-5 i.a.i.d.b.o.BigQueryDirectLoadingStorageOperation(writeToStage):66 Writing data to table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte_internal, tableId=jira_airbyte_raw__stream_jira_issues_onlyissues_airbyte_tmp}} with schema Schema{fields=[Field{name=_airbyte_raw_id, type=STRING, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_extracted_at, type=TIMESTAMP, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_loaded_at, type=TIMESTAMP, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_data, type=STRING, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_meta, type=STRING, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_generation_id, type=INTEGER, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}]}
2025-04-15 11:06:14 destination INFO pool-4-thread-5 i.a.i.d.b.o.BigQueryDirectLoadingStorageOperation(writeToStage):79 Writing to channel completed for GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte_internal, tableId=jira_airbyte_raw__stream_jira_issues_onlyissues_airbyte_tmp}}
2025-04-15 11:06:14 destination WARN pool-4-thread-5 i.a.i.d.b.BigQueryUtils(waitForJobFinish):292 Received null value for Job, nothing to waitFor
2025-04-15 11:06:14 destination INFO pool-4-thread-5 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states
2025-04-15 11:06:14 destination INFO pool-4-thread-5 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete
2025-04-15 11:06:14 destination INFO pool-4-thread-5 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2025-04-15 11:06:14 destination INFO pool-4-thread-5 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2025-04-15 11:06:14 destination INFO pool-4-thread-5 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (b09a1) -- Worker finished flushing. Current queue size: 0
2025-04-15 11:06:54 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 3 GB, allocated: 9.93 MB (9.926865577697754 MB), %% used: 0.0032314015552401543 | Queue `jira_issues_onlyissues`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 9 MB, Used: -76687 bytes, percentage Used -0.007367322719323805
2025-04-15 11:06:54 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-04-15 11:07:54 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 3 GB, allocated: 9.93 MB (9.926865577697754 MB), %% used: 0.0032314015552401543 | Queue `jira_issues_onlyissues`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 9 MB, Used: -76687 bytes, percentage Used -0.007367322719323805
2025-04-15 11:07:54 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-04-15 11:08:33 platform INFO Canceling workload 5f6bdf23-5fd6-4da6-beba-08ce186b5b9e_17038_0_sync
2025-04-15 11:08:36 replication-orchestrator WARN Cancelling sync, workload is in a terminal state
2025-04-15 11:08:36 replication-orchestrator INFO processMessage: done. (fromSource.isDone:false, forDest.isClosed:false)
2025-04-15 11:08:36 replication-orchestrator INFO writeToDestination: exception caught
2025-04-15 11:08:36 replication-orchestrator INFO writeToDestination: done. (forDest.isDone:true, isDestRunning:true)
2025-04-15 11:08:36 destination INFO main i.a.c.i.b.IntegrationRunner$Companion(consumeWriteStream$io_airbyte_airbyte_cdk_java_airbyte_cdk_airbyte_cdk_core):428 Finished buffered read of input stream
2025-04-15 11:08:36 destination INFO main i.a.c.i.d.a.FlushWorkers(close):193 Closing flush workers -- waiting for all buffers to flush
2025-04-15 11:08:36 destination INFO main i.a.c.i.d.a.FlushWorkers(close):226 REMAINING_BUFFERS_INFO
Namespace: jira_airbyte Stream: jira_issues_onlyissues -- remaining records: 1
2025-04-15 11:08:36 destination INFO main i.a.c.i.d.a.FlushWorkers(close):227 Waiting for all streams to flush.
2025-04-15 11:08:36 destination INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):109 flushing: trigger info: jira_airbyte - jira_issues_onlyissues, time trigger: false , size trigger: true current threshold b: 0 bytes, queue size b: 45.61 KB, penalty b: 0 bytes, after penalty b: 45.61 KB
2025-04-15 11:08:36 destination INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):135 Flush Worker (edb26) -- Worker picked up work.
2025-04-15 11:08:36 destination INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):141 Flush Worker (edb26) -- Attempting to read from queue namespace: jira_airbyte, stream: jira_issues_onlyissues.
2025-04-15 11:08:36 destination INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 10439060 bytes..
2025-04-15 11:08:36 destination INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):159 Flush Worker (edb26) -- Batch contains: 1 records, 45.61 KB bytes.
```

### Contribute

- [ ] Yes, I want to contribute

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.