[BigQuery - smartico.ai ] is not working for big querys
- Linguagem predominante
- Python
- Estrelas
- 22.1k
- Forks
- 5.4k
- Merge médio
- 5h
- PRs com merge (30d)
- 671
Descrição
### Connector Name
BigQuery - smartico.ai
### Connector Version
0.4.5
### What step the error happened?
During the sync
### Relevant information
Issue: The BigQuery source connector (version 0.4.5) is failing to sync table `dwh_ext_28174.j_engagements` due to BigQuery API response size limits. The error "Response too large to return. Consider specifying a destination table in your job configuration" occurs consistently across multiple sync attempts.
Customer Language: Portuguese (Brasil)
Connection Details:
- Workspace ID: [redacted]
- Connection ID: [redacted]
- Connection Name: "BigQuery → Snowflake"
- Source ID: [redacted] (BigQuery)
- Destination ID: [redacted] (Snowflake)
- Source Connector: source-bigquery v0.4.5
- Failed Job ID: 101186835 (multiple attempts)
Problem: The sync fails during the read operation from BigQuery. The BigQuery API returns an error indicating the response is too large. This is a known limitation where BigQuery refuses to return very large result sets without using a destination table configuration. The current connector version does not support this BigQuery feature.
Root Cause: The table `dwh_ext_28174.j_engagements` contains too much data for BigQuery to return directly via API. The source-bigquery connector v0.4.5 does not implement the workaround of using a destination table in the job configuration, which is what BigQuery suggests in the error message.
Verified Data:
- Connection tests pass successfully for both source and destination
- Other tables may sync successfully (only j_engagements is confirmed failing)
- User has confirmed the sync failed 3 times with the same error
- User authorized workspace access for investigation
Impact: The user cannot sync the `j_engagements` table, which is a critical data source. The user does NOT have access to the BigQuery instance to create filtered views or modify queries as a workaround. This blocks their entire data pipeline for this table.
Requested Fix:
1. Immediate: Investigate if there's a configuration or workaround available for the current connector version
2. Long-term: Implement support for BigQuery's destination table configuration to handle large result sets
3. Alternative: Provide guidance on any other solutions that don't require BigQuery access
Additional context: The user is working in Portuguese and does not have administrative access to modify the source BigQuery environment.
### Relevant log output
```shell
Recent Sync Logs: Key error from job 101186835, attempt 1:
ERROR i.a.i.b.BigQuerySource(lambda$getServerDataIterator$6):232 Failed to retrieve data from BigQuery for stream: j_engagements
com.google.cloud.bigquery.BigQueryException: Response too large to return. Consider specifying a destination table in your job configuration. For more details, see https://cloud.google.com/bigquery/docs/writing-results
```
### Contribute
- [ ] Yes, I want to contribute
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13375
Guia de contribuição
Direção de pesquisa
Start in the source-bigquery connector at the getServerDataIterator path referenced by the log (line 232), and reproduce the failure with a result set that exceeds BigQuery's response limit. Review how the connector configures read jobs and verify the sync completes for j_engagements or an equivalent large table without the response-too-large error.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- java
- Domínio
- data-engineering, databases
- Tipo de issue
- Bug
- Dificuldade
- 4/5
- Tempo estimado
- 3-5 dias
- Status de atividade
- Ativa
- Clareza
- Razoavelmente clara
- Facilidade para iniciantes
- 45/100