firebase / firebase/extensions
🐛 [firestore-export-bigquery] Error initializing BigQuery resources when running fs-bq-import-collection
- Dominant language
- TypeScript
- Stars
- 979
- Forks
- 433
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 125
Description
- Extension name: firestore-bigquery-export
- Extension version: 0.2.5
- Configuration values (redact info where appropriate):
- VIEW_TYPE = "materialized_incremental"
- CLUSTERING = "timestamp"
Yet the issue is rather in the `import` script from that repo, bundeled in npm package `@firebaseextensions/fs-bq-import-collection@0.1.24`.
### [REQUIRED] Step 3: Describe the problem
#### Steps to reproduce:
Deploy the extension firestore-bigquery-export using the view type `materialized_incremental` recently introduced. Bonus: also specify "timestamp" as `clustering` field.
Then run the backfill command using this command:
```
npx @firebaseextensions/fs-bq-import-collection \
--non-interactive \
--project=xxx \
--big-query-project=xxx \
--source-collection-path=my_collection \
--query-collection-group=false \
--dataset=firestore_export \
--dataset-location=europe-west1 \
--table-name-prefix=my_collection
```
##### Expected result
I expect the command **not to update** my existing BigQuey tables / views, but simply to `INSERT` new rows with any document data found in my Firestore collection.
##### Actual result
The script drops my clustering, for no reason, it would also have altered my partitionning if I had one.
The finally script crashes with this error:
```
{"severity":"INFO","message":"BigQuery dataset already exists: firestore_export"}
{"severity":"INFO","message":"Clustering removed on my_collection_raw_changelog"}
{"severity":"WARNING","message":"Did not add partitioning to schema: Partitioning not enabled"}
{"severity":"INFO","message":"Updated Metadata on my_collection_raw_changelog, {\"config\":{\"tableId\":\"my_collection\",\"datasetId\":\"firestore_export\",\"datasetLocation\":\"europe-west1\",\"wildcardIds\":false,\"useNewSnapshotQuerySyntax\":false,\"bqProjectId\":\"xxx\",\"firestoreInstanceId\":\"(default)\"},\"documentIdColExists\":{\"name\":\"document_id\",\"type\":\"STRING\",\"mode\":\"NULLABLE\",\"description\":\"The document id as defined in the firestore database.\"},\"oldDataColExists\":{\"name\":\"old_data\",\"type\":\"STRING\",\"mode\":\"NULLABLE\",\"description\":\"The full JSON representation of the document state before the indicated operation is applied. This field will be null for CREATE operations.\"}})"}
{"severity":"INFO","message":"View with id my_collection_raw_latest already exists in dataset firestore_export."}
Error initializing BigQuery resources: Error initializing latest view: Cannot set a view definition for xxx:firestore_export.my_collection_raw_latest because it is not of type View.
Error importing Collection to BigQuery: Error: Error initializing latest view: Cannot set a view definition for xxx:firestore_export.my_collection_raw_latest because it is not of type View.
```
Contributor guide
Assessment
This issue has not been assessed yet.