[BigQuery - smartico.ai ] is not working for big querys
- Lingua principale
- Python
- Stelle
- 22.1k
- Fork
- 5.3k
- Metriche di merge delle PR
- Metriche PR in attesa
Descrizione
### Connector Name
BigQuery - smartico.ai
### Connector Version
0.4.5
### What step the error happened?
During the sync
### Relevant information
Issue: The BigQuery source connector (version 0.4.5) is failing to sync table `dwh_ext_28174.j_engagements` due to BigQuery API response size limits. The error "Response too large to return. Consider specifying a destination table in your job configuration" occurs consistently across multiple sync attempts.
Customer Language: Portuguese (Brasil)
Connection Details:
- Workspace ID: [redacted]
- Connection ID: [redacted]
- Connection Name: "BigQuery → Snowflake"
- Source ID: [redacted] (BigQuery)
- Destination ID: [redacted] (Snowflake)
- Source Connector: source-bigquery v0.4.5
- Failed Job ID: 101186835 (multiple attempts)
Problem: The sync fails during the read operation from BigQuery. The BigQuery API returns an error indicating the response is too large. This is a known limitation where BigQuery refuses to return very large result sets without using a destination table configuration. The current connector version does not support this BigQuery feature.
Root Cause: The table `dwh_ext_28174.j_engagements` contains too much data for BigQuery to return directly via API. The source-bigquery connector v0.4.5 does not implement the workaround of using a destination table in the job configuration, which is what BigQuery suggests in the error message.
Verified Data:
- Connection tests pass successfully for both source and destination
- Other tables may sync successfully (only j_engagements is confirmed failing)
- User has confirmed the sync failed 3 times with the same error
- User authorized workspace access for investigation
Impact: The user cannot sync the `j_engagements` table, which is a critical data source. The user does NOT have access to the BigQuery instance to create filtered views or modify queries as a workaround. This blocks their entire data pipeline for this table.
Requested Fix:
1. Immediate: Investigate if there's a configuration or workaround available for the current connector version
2. Long-term: Implement support for BigQuery's destination table configuration to handle large result sets
3. Alternative: Provide guidance on any other solutions that don't require BigQuery access
Additional context: The user is working in Portuguese and does not have administrative access to modify the source BigQuery environment.
### Relevant log output
```shell
Recent Sync Logs: Key error from job 101186835, attempt 1:
ERROR i.a.i.b.BigQuerySource(lambda$getServerDataIterator$6):232 Failed to retrieve data from BigQuery for stream: j_engagements
com.google.cloud.bigquery.BigQueryException: Response too large to return. Consider specifying a destination table in your job configuration. For more details, see https://cloud.google.com/bigquery/docs/writing-results
```
### Contribute
- [ ] Yes, I want to contribute
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13375
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.