airbytehq / airbytehq/airbyte

[BigQuery - smartico.ai ] is not working for big querys

Aberta
#84,978 3 comentários 0 reações 0 responsáveis Ver no GitHub
area/connectors autoteam community connectors/source/bigquery needs-triage team/extensibility team/use type/bug
Linguagem predominante
Python
Estrelas
22.1k
Forks
5.4k
Merge médio
5h
PRs com merge (30d)
671

Descrição

### Connector Name

BigQuery - smartico.ai

### Connector Version

0.4.5

### What step the error happened?

During the sync

### Relevant information

Issue: The BigQuery source connector (version 0.4.5) is failing to sync table `dwh_ext_28174.j_engagements` due to BigQuery API response size limits. The error "Response too large to return. Consider specifying a destination table in your job configuration" occurs consistently across multiple sync attempts.

Customer Language: Portuguese (Brasil)

Connection Details:
- Workspace ID: [redacted]
- Connection ID: [redacted]
- Connection Name: "BigQuery → Snowflake"
- Source ID: [redacted] (BigQuery)
- Destination ID: [redacted] (Snowflake)
- Source Connector: source-bigquery v0.4.5
- Failed Job ID: 101186835 (multiple attempts)

Problem: The sync fails during the read operation from BigQuery. The BigQuery API returns an error indicating the response is too large. This is a known limitation where BigQuery refuses to return very large result sets without using a destination table configuration. The current connector version does not support this BigQuery feature.

Root Cause: The table `dwh_ext_28174.j_engagements` contains too much data for BigQuery to return directly via API. The source-bigquery connector v0.4.5 does not implement the workaround of using a destination table in the job configuration, which is what BigQuery suggests in the error message.

Verified Data:
- Connection tests pass successfully for both source and destination
- Other tables may sync successfully (only j_engagements is confirmed failing)
- User has confirmed the sync failed 3 times with the same error
- User authorized workspace access for investigation

Impact: The user cannot sync the `j_engagements` table, which is a critical data source. The user does NOT have access to the BigQuery instance to create filtered views or modify queries as a workaround. This blocks their entire data pipeline for this table.

Requested Fix:
1. Immediate: Investigate if there's a configuration or workaround available for the current connector version
2. Long-term: Implement support for BigQuery's destination table configuration to handle large result sets
3. Alternative: Provide guidance on any other solutions that don't require BigQuery access

Additional context: The user is working in Portuguese and does not have administrative access to modify the source BigQuery environment.

### Relevant log output

```shell
Recent Sync Logs: Key error from job 101186835, attempt 1:

ERROR i.a.i.b.BigQuerySource(lambda$getServerDataIterator$6):232 Failed to retrieve data from BigQuery for stream: j_engagements
com.google.cloud.bigquery.BigQueryException: Response too large to return. Consider specifying a destination table in your job configuration. For more details, see https://cloud.google.com/bigquery/docs/writing-results
```

### Contribute

- [ ] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13375

Guia de contribuição

Abrir o guia de contribuição

Direção de pesquisa

Start in the source-bigquery connector at the getServerDataIterator path referenced by the log (line 232), and reproduce the failure with a result set that exceeds BigQuery's response limit. Review how the connector configures read jobs and verify the sync completes for j_engagements or an equivalent large table without the response-too-large error.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
java
Domínio
data-engineering, databases
Tipo de issue
Bug
Dificuldade
4/5
Tempo estimado
3-5 dias
Status de atividade
Ativa
Clareza
Razoavelmente clara
Facilidade para iniciantes
45/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.