[source-hubspot] [destination-bigquery] Issue: Incremental sync for line_items stalls when syncing to Big Query
- Lingua principale
- Python
- Stelle
- 22.1k
- Fork
- 5.3k
- Metriche di merge delle PR
- Metriche PR in attesa
Descrizione
### Connector Name
source-hubspot destination-bigquery
### Connector Version
4.4.9 - 2.9.2
### What step the error happened?
During the sync
### Relevant information
**Issue Summary**
I am experiencing a significant delay when syncing line_items from HubSpot to BigQuery using Airbyte. The process takes an excessive amount of time to start, even though the number of records is small. Logs indicate that the sync remains idle for at least an hour before actually beginning.
**Key Observations**
Other HubSpot tables sync without issues – This problem is specific to line_items.
Sync hangs before it officially starts – The workload is created and marked as RUNNING, but no actual data transfer occurs for a long time.
The logs repeatedly show idle messages, such as:
Pool queue size: 0, Active threads: 0
This suggests that the process is not actively fetching or writing data.
The buffer remains empty, as indicated by:
Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
No explicit errors appear in the logs, meaning there is no immediate failure but rather a stall in execution.
[hs_bq_line_items_logs.txt](https://github.com/user-attachments/files/18690099/hs_bq_line_items_logs.txt)
**Possible Causes Considered**
API rate limits? → Unlikely, since other tables sync fine.
Permissions? → Not an issue, as full refresh works.
Schema-related issues? → There might be a problem with how line_items is processed.
Pagination or cursor (updatedAt) issues? → Airbyte may be stuck waiting for changes.
**Questions for Support**
Is there a known issue with syncing line_items from HubSpot?
Could this be related to how Airbyte handles incremental syncs on deeply nested or structured data?
Are there specific configurations or workarounds to speed up incremental syncs for this stream?
**Logs**
I have attached the full logs for further investigation. Let me know if you need additional information.
[line_items_fresh_test_logs_192_txt.txt](https://github.com/user-attachments/files/18767495/line_items_fresh_test_logs_192_txt.txt)
Thanks for your help! 🚀
### Relevant log output
```shell
2025-02-11 20:37:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:38:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:38:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:39:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:39:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:40:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:40:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:41:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:41:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:42:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:42:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:43:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:43:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:44:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:44:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:45:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:45:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:46:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:46:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-11 20:47:26 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-11 20:47:26 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
```
### Contribute
- [x] Yes, I want to contribute
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.