airbytehq / airbytehq/airbyte

[BigQuery - smartico.ai ] is not working for big querys

Aperta
#84,978 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub
area/connectors autoteam community connectors/source/bigquery needs-triage team/extensibility team/use type/bug
Lingua principale
Python
Stelle
22.1k
Fork
5.3k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

### Connector Name

BigQuery - smartico.ai

### Connector Version

0.4.5

### What step the error happened?

During the sync

### Relevant information

Issue: The BigQuery source connector (version 0.4.5) is failing to sync table `dwh_ext_28174.j_engagements` due to BigQuery API response size limits. The error "Response too large to return. Consider specifying a destination table in your job configuration" occurs consistently across multiple sync attempts.

Customer Language: Portuguese (Brasil)

Connection Details:
- Workspace ID: [redacted]
- Connection ID: [redacted]
- Connection Name: "BigQuery → Snowflake"
- Source ID: [redacted] (BigQuery)
- Destination ID: [redacted] (Snowflake)
- Source Connector: source-bigquery v0.4.5
- Failed Job ID: 101186835 (multiple attempts)

Problem: The sync fails during the read operation from BigQuery. The BigQuery API returns an error indicating the response is too large. This is a known limitation where BigQuery refuses to return very large result sets without using a destination table configuration. The current connector version does not support this BigQuery feature.

Root Cause: The table `dwh_ext_28174.j_engagements` contains too much data for BigQuery to return directly via API. The source-bigquery connector v0.4.5 does not implement the workaround of using a destination table in the job configuration, which is what BigQuery suggests in the error message.

Verified Data:
- Connection tests pass successfully for both source and destination
- Other tables may sync successfully (only j_engagements is confirmed failing)
- User has confirmed the sync failed 3 times with the same error
- User authorized workspace access for investigation

Impact: The user cannot sync the `j_engagements` table, which is a critical data source. The user does NOT have access to the BigQuery instance to create filtered views or modify queries as a workaround. This blocks their entire data pipeline for this table.

Requested Fix:
1. Immediate: Investigate if there's a configuration or workaround available for the current connector version
2. Long-term: Implement support for BigQuery's destination table configuration to handle large result sets
3. Alternative: Provide guidance on any other solutions that don't require BigQuery access

Additional context: The user is working in Portuguese and does not have administrative access to modify the source BigQuery environment.

### Relevant log output

```shell
Recent Sync Logs: Key error from job 101186835, attempt 1:

ERROR i.a.i.b.BigQuerySource(lambda$getServerDataIterator$6):232 Failed to retrieve data from BigQuery for stream: j_engagements
com.google.cloud.bigquery.BigQueryException: Response too large to return. Consider specifying a destination table in your job configuration. For more details, see https://cloud.google.com/bigquery/docs/writing-results
```

### Contribute

- [ ] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13375

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.