airbytehq / airbytehq/airbyte

[source-hubspot] [destination-bigquery] Issue: Incremental sync for line_items stalls when syncing to Big Query

Aperta
#53,199 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub
area/connectors community connectors/source/hubspot team/extensibility type/bug
Lingua principale
Python
Stelle
22.1k
Fork
5.3k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

### Connector Name

source-hubspot destination-bigquery

### Connector Version

4.4.9 - 2.9.2

### What step the error happened?

During the sync

### Relevant information

**Issue Summary**
I am experiencing a significant delay when syncing line_items from HubSpot to BigQuery using Airbyte. The process takes an excessive amount of time to start, even though the number of records is small. Logs indicate that the sync remains idle for at least an hour before actually beginning.

**Key Observations**
Other HubSpot tables sync without issues – This problem is specific to line_items.
Sync hangs before it officially starts – The workload is created and marked as RUNNING, but no actual data transfer occurs for a long time.
The logs repeatedly show idle messages, such as:

Pool queue size: 0, Active threads: 0

This suggests that the process is not actively fetching or writing data.
The buffer remains empty, as indicated by:

Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0

No explicit errors appear in the logs, meaning there is no immediate failure but rather a stall in execution.

[hs_bq_line_items_logs.txt](https://github.com/user-attachments/files/18690099/hs_bq_line_items_logs.txt)

**Possible Causes Considered**
API rate limits? → Unlikely, since other tables sync fine.
Permissions? → Not an issue, as full refresh works.
Schema-related issues? → There might be a problem with how line_items is processed.
Pagination or cursor (updatedAt) issues? → Airbyte may be stuck waiting for changes.

**Questions for Support**
Is there a known issue with syncing line_items from HubSpot?
Could this be related to how Airbyte handles incremental syncs on deeply nested or structured data?
Are there specific configurations or workarounds to speed up incremental syncs for this stream?

**Logs**
I have attached the full logs for further investigation. Let me know if you need additional information.

[line_items_fresh_test_logs_192_txt.txt](https://github.com/user-attachments/files/18767495/line_items_fresh_test_logs_192_txt.txt)

Thanks for your help! 🚀

### Relevant log output

```shell
2025-02-11 20:37:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:38:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:38:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:39:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:39:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:40:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:40:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:41:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:41:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:42:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:42:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:43:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:43:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:44:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:44:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:45:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:45:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:46:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:46:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:47:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:47:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
```

### Contribute

- [x] Yes, I want to contribute

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.