firebase / firebase/extensions
🐛 [firestore-bigquery-export] Task size too large errors occuring even with EXCLUDE_OLD_DATA set to yes/true
- Dominant language
- TypeScript
- Stars
- 979
- Forks
- 433
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 125
Description
### [READ] Step 1: Are you in the right place?
Issues filed here should be about bugs for a **specific extension in this repository**.
If you have a general question, need help debugging, or fall into some
other category use one of these other channels:
- For general technical questions, post a question on [StackOverflow](http://stackoverflow.com/)
with the firebase tag.
- For general Firebase discussion, use the [firebase-talk](https://groups.google.com/forum/#!forum/firebase-talk)
google group.
- To file a bug against the Firebase Extensions platform, or for an issue affecting multiple extensions, please reach out to
[Firebase support](https://firebase.google.com/support/troubleshooter/contact/) directly.
### [REQUIRED] Step 2: Describe your configuration
- Extension name: **firestore-bigquery-export**
- Extension version: **0.1.50**
- Configuration values (redact info where appropriate):
- BigQuery Dataset location: us
- BigQuery Project ID: xxx
- Database ID: (default)
- Collection path: xxx
- Enable Wildcard Column field with Parent Firestore Document IDs (Optional): false
- Dataset ID: xxx
- Table ID: xxx
- BigQuery SQL table Time Partitioning option type (Optional): DAY
- BigQuery Time Partitioning column name (Optional): timestamp
- Firestore Document field name for BigQuery SQL Time Partitioning field option (Optional): Parameter not set
- BigQuery SQL Time Partitioning table schema field(column) type (Optional): omit
- BigQuery SQL table clustering (Optional): document_id
- Maximum number of synced documents per second (Optional): 100
- Backup Collection Name (Optional): Parameter not set
- Transform function URL (Optional): Parameter not set
- Use new query syntax for snapshots: no
- Exclude old data payloads (Optional): yes
- Use Collection Group query (Optional): no
- Cloud KMS key name (Optional): Parameter not set
### [REQUIRED] Step 3: Describe the problem
Even when using the `EXCLUDE_OLD_DATA` setting to prevent old_data from being populated, we are still seeing `Task size too large` errors on many messages. This might mean that firestore payloads close to 1MB are being padded in a way that the subsequent Task exceeds 1MB
#### Steps to reproduce:
1. Install extension version 0.1.50, ensure that `Exclude old data payloads` is set to `yes`
2. Write a large document to firestore
##### Expected result
No `Task size too large` errors should appear at all.
##### Actual result
Observing many `Task size too large` errors in logs.
Contributor guide
Assessment
This issue has not been assessed yet.