[destination-bigquery] 10x Performance Regression from 2.11.0 versions on large data volumes
- Langage dominant
- Python
- Étoiles
- 22.1k
- Forks
- 5.3k
- Métriques de merge des PR
- Métriques de PR en attente
Description
### Connector Name
destination-bigquery
### Connector Version
3.0.13
### What step the error happened?
During the sync
### Relevant information
We were previously using the BigQuery destination (`2.9.2`) with the Google Cloud Storage source (`0.10.4`) to load data from Cloud Storage to BigQuery. The destination is configured to use the GCS Staging loading method with "Disable Final Tables" enabled (loading raw data only).
We recently updated the destination version to `3.0.13, and the load time for large datasets has increased by more than 10x (from 8 minutes to 1 hour 40 minutes for 914.63 MB (553,180 records). Small datasets continue to work as expected, and we have not observed issues with other source types.
After some investigation, I have figured out that this issue appeared in `2.11.0-rc.1` BigQuery destination version, and the last version with which everything works as previously is `2.10.2`.
The sync performance degrades in a specific pattern:
* The sync runs normally until 90,000 records (144 MB).
* It then hangs/stalls for over 4 minutes.
* It resumes and loads another batch up to roughly 140,000 records (225 MB).
* It hangs again for 5 minutes.
* The pattern repeats, but the batch size processed between stalls appears to decrease over time.
We attempted to increase the resources (CPU/RAM) for the jobs, but this did not resolve the issue.
### Relevant log output
```shell
2025-11-13 05:27:02 replication-orchestrator INFO Records read: 5000 (8 MB)
2025-11-13 05:27:06 replication-orchestrator INFO Records read: 10000 (16 MB)
2025-11-13 05:27:11 replication-orchestrator INFO Records read: 15000 (24 MB)
2025-11-13 05:27:15 replication-orchestrator INFO Records read: 20000 (32 MB)
2025-11-13 05:27:21 replication-orchestrator INFO Records read: 25000 (40 MB)
2025-11-13 05:27:25 replication-orchestrator INFO Records read: 30000 (48 MB)
2025-11-13 05:27:30 replication-orchestrator INFO Records read: 35000 (56 MB)
2025-11-13 05:27:35 replication-orchestrator INFO Records read: 40000 (64 MB)
2025-11-13 05:27:40 replication-orchestrator INFO Records read: 45000 (72 MB)
2025-11-13 05:27:44 replication-orchestrator INFO Records read: 50000 (80 MB)
2025-11-13 05:27:49 replication-orchestrator INFO Records read: 55000 (88 MB)
2025-11-13 05:27:54 replication-orchestrator INFO Records read: 60000 (96 MB)
2025-11-13 05:27:59 replication-orchestrator INFO Records read: 65000 (104 MB)
2025-11-13 05:28:03 replication-orchestrator INFO Records read: 70000 (112 MB)
2025-11-13 05:28:06 replication-orchestrator INFO Records read: 75000 (120 MB)
2025-11-13 05:28:10 replication-orchestrator INFO Records read: 80000 (128 MB)
2025-11-13 05:28:14 replication-orchestrator INFO Records read: 85000 (136 MB)
2025-11-13 05:28:18 replication-orchestrator INFO Records read: 90000 (144 MB)
2025-11-13 05:32:39 replication-orchestrator INFO Records read: 95000 (152 MB)
2025-11-13 05:32:43 replication-orchestrator INFO Records read: 100000 (161 MB)
2025-11-13 05:32:47 replication-orchestrator INFO Records read: 105000 (169 MB)
2025-11-13 05:32:51 replication-orchestrator INFO Records read: 110000 (177 MB)
2025-11-13 05:32:54 replication-orchestrator INFO Records read: 115000 (185 MB)
2025-11-13 05:32:59 replication-orchestrator INFO Records read: 120000 (193 MB)
2025-11-13 05:33:02 replication-orchestrator INFO Records read: 125000 (201 MB)
2025-11-13 05:33:05 replication-orchestrator INFO Records read: 130000 (209 MB)
2025-11-13 05:33:09 replication-orchestrator INFO Records read: 135000 (217 MB)
2025-11-13 05:33:13 replication-orchestrator INFO Records read: 140000 (225 MB)
2025-11-13 05:38:35 replication-orchestrator INFO Records read: 145000 (233 MB)
2025-11-13 05:38:39 replication-orchestrator INFO Records read: 150000 (241 MB)
2025-11-13 05:38:42 replication-orchestrator INFO Records read: 155000 (249 MB)
2025-11-13 05:38:45 replication-orchestrator INFO Records read: 160000 (257 MB)
2025-11-13 05:38:48 replication-orchestrator INFO Records read: 165000 (266 MB)
2025-11-13 05:38:52 replication-orchestrator INFO Records read: 170000 (274 MB)
2025-11-13 05:38:56 replication-orchestrator INFO Records read: 175000 (282 MB)
2025-11-13 05:38:59 replication-orchestrator INFO Records read: 180000 (290 MB)
2025-11-13 05:39:02 replication-orchestrator INFO Records read: 185000 (298 MB)
2025-11-13 05:44:36 replication-orchestrator INFO Records read: 190000 (306 MB)
2025-11-13 05:44:39 replication-orchestrator INFO Records read: 195000 (314 MB)
2025-11-13 05:44:43 replication-orchestrator INFO Records read: 200000 (322 MB)
2025-11-13 05:44:47 replication-orchestrator INFO Records read: 205000 (331 MB)
2025-11-13 05:44:51 replication-orchestrator INFO Records read: 210000 (339 MB)
2025-11-13 05:44:54 replication-orchestrator INFO Records read: 215000 (347 MB)
2025-11-13 05:44:57 replication-orchestrator INFO Records read: 220000 (355 MB)
2025-11-13 05:50:35 replication-orchestrator INFO Records read: 225000 (363 MB)
2025-11-13 05:50:38 replication-orchestrator INFO Records read: 230000 (371 MB)
2025-11-13 05:50:42 replication-orchestrator INFO Records read: 235000 (379 MB)
2025-11-13 05:50:46 replication-orchestrator INFO Records read: 240000 (387 MB)
2025-11-13 05:50:50 replication-orchestrator INFO Records read: 245000 (395 MB)
2025-11-13 05:50:54 replication-orchestrator INFO Records read: 250000 (404 MB)
2025-11-13 05:56:33 replication-orchestrator INFO Records read: 255000 (412 MB)
2025-11-13 05:56:37 replication-orchestrator INFO Records read: 260000 (420 MB)
2025-11-13 05:56:41 replication-orchestrator INFO Records read: 265000 (428 MB)
2025-11-13 05:56:45 replication-orchestrator INFO Records read: 270000 (436 MB)
2025-11-13 05:56:49 replication-orchestrator INFO Records read: 275000 (444 MB)
2025-11-13 05:56:53 replication-orchestrator INFO Records read: 280000 (452 MB)
2025-11-13 06:02:35 replication-orchestrator INFO Records read: 285000 (460 MB)
2025-11-13 06:02:39 replication-orchestrator INFO Records read: 290000 (468 MB)
2025-11-13 06:02:43 replication-orchestrator INFO Records read: 295000 (476 MB)
2025-11-13 06:02:47 replication-orchestrator INFO Records read: 300000 (485 MB)
2025-11-13 06:02:49 replication-orchestrator INFO Records read: 305000 (493 MB)
2025-11-13 06:02:51 replication-orchestrator INFO Records read: 310000 (501 MB)
2025-11-13 06:02:52 replication-orchestrator INFO Records read: 315000 (510 MB)
2025-11-13 06:02:55 replication-orchestrator INFO Records read: 320000 (518 MB)
2025-11-13 06:08:35 replication-orchestrator INFO Records read: 325000 (527 MB)
2025-11-13 06:08:40 replication-orchestrator INFO Records read: 330000 (535 MB)
2025-11-13 06:08:44 replication-orchestrator INFO Records read: 335000 (543 MB)
2025-11-13 06:08:46 replication-orchestrator INFO Records read: 340000 (551 MB)
2025-11-13 06:08:48 replication-orchestrator INFO Records read: 345000 (560 MB)
2025-11-13 06:08:50 replication-orchestrator INFO Records read: 350000 (568 MB)
2025-11-13 06:08:52 replication-orchestrator INFO Records read: 355000 (577 MB)
2025-11-13 06:08:54 replication-orchestrator INFO Records read: 360000 (585 MB)
2025-11-13 06:14:34 replication-orchestrator INFO Records read: 365000 (594 MB)
2025-11-13 06:14:38 replication-orchestrator INFO Records read: 370000 (602 MB)
2025-11-13 06:14:42 replication-orchestrator INFO Records read: 375000 (610 MB)
2025-11-13 06:14:45 replication-orchestrator INFO Records read: 380000 (618 MB)
2025-11-13 06:14:49 replication-orchestrator INFO Records read: 385000 (627 MB)
2025-11-13 06:20:32 replication-orchestrator INFO Records read: 390000 (635 MB)
2025-11-13 06:20:36 replication-orchestrator INFO Records read: 395000 (643 MB)
2025-11-13 06:20:42 replication-orchestrator INFO Records read: 400000 (651 MB)
2025-11-13 06:20:46 replication-orchestrator INFO Records read: 405000 (659 MB)
2025-11-13 06:27:06 replication-orchestrator INFO Records read: 410000 (667 MB)
2025-11-13 06:27:12 replication-orchestrator INFO Records read: 415000 (676 MB)
2025-11-13 06:27:17 replication-orchestrator INFO Records read: 420000 (684 MB)
2025-11-13 06:32:34 replication-orchestrator INFO Records read: 425000 (692 MB)
2025-11-13 06:32:40 replication-orchestrator INFO Records read: 430000 (700 MB)
2025-11-13 06:32:45 replication-orchestrator INFO Records read: 435000 (708 MB)
2025-11-13 06:38:36 replication-orchestrator INFO Records read: 440000 (716 MB)
2025-11-13 06:38:43 replication-orchestrator INFO Records read: 445000 (725 MB)
2025-11-13 06:44:32 replication-orchestrator INFO Records read: 450000 (733 MB)
2025-11-13 06:44:37 replication-orchestrator INFO Records read: 455000 (741 MB)
2025-11-13 06:44:39 replication-orchestrator INFO Records read: 460000 (749 MB)
2025-11-13 06:44:44 replication-orchestrator INFO Records read: 465000 (758 MB)
2025-11-13 06:50:36 replication-orchestrator INFO Records read: 470000 (766 MB)
2025-11-13 06:50:41 replication-orchestrator INFO Records read: 475000 (774 MB)
2025-11-13 06:56:33 replication-orchestrator INFO Records read: 480000 (782 MB)
2025-11-13 06:56:40 replication-orchestrator INFO Records read: 485000 (790 MB)
2025-11-13 07:02:33 replication-orchestrator INFO Records read: 490000 (798 MB)
```
### Contribute
- [ ] Yes, I want to contribute
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.