firebase / firebase/extensions

🐛 [firestore-export-bigquery] Error initializing BigQuery resources when running fs-bq-import-collection

Open
#2,468 3 comments 0 reactions 2 assignees Claimed by @IzaakGough View on GitHub
reproduced type: bug
Dominant language
TypeScript
Stars
979
Forks
433
Avg merge
1d 13h
Merged PRs (30d)
125

Description

- Extension name: firestore-bigquery-export
- Extension version: 0.2.5
- Configuration values (redact info where appropriate):
- VIEW_TYPE = "materialized_incremental"
- CLUSTERING = "timestamp"

Yet the issue is rather in the `import` script from that repo, bundeled in npm package `@firebaseextensions/fs-bq-import-collection@0.1.24`.

### [REQUIRED] Step 3: Describe the problem

#### Steps to reproduce:

Deploy the extension firestore-bigquery-export using the view type `materialized_incremental` recently introduced. Bonus: also specify "timestamp" as `clustering` field.

Then run the backfill command using this command:

```
npx @firebaseextensions/fs-bq-import-collection \
--non-interactive \
--project=xxx \
--big-query-project=xxx \
--source-collection-path=my_collection \
--query-collection-group=false \
--dataset=firestore_export \
--dataset-location=europe-west1 \
--table-name-prefix=my_collection
```

##### Expected result

I expect the command **not to update** my existing BigQuey tables / views, but simply to `INSERT` new rows with any document data found in my Firestore collection.

##### Actual result

The script drops my clustering, for no reason, it would also have altered my partitionning if I had one.

The finally script crashes with this error:

```
{"severity":"INFO","message":"BigQuery dataset already exists: firestore_export"}
{"severity":"INFO","message":"Clustering removed on my_collection_raw_changelog"}
{"severity":"WARNING","message":"Did not add partitioning to schema: Partitioning not enabled"}
{"severity":"INFO","message":"Updated Metadata on my_collection_raw_changelog, {\"config\":{\"tableId\":\"my_collection\",\"datasetId\":\"firestore_export\",\"datasetLocation\":\"europe-west1\",\"wildcardIds\":false,\"useNewSnapshotQuerySyntax\":false,\"bqProjectId\":\"xxx\",\"firestoreInstanceId\":\"(default)\"},\"documentIdColExists\":{\"name\":\"document_id\",\"type\":\"STRING\",\"mode\":\"NULLABLE\",\"description\":\"The document id as defined in the firestore database.\"},\"oldDataColExists\":{\"name\":\"old_data\",\"type\":\"STRING\",\"mode\":\"NULLABLE\",\"description\":\"The full JSON representation of the document state before the indicated operation is applied. This field will be null for CREATE operations.\"}})"}
{"severity":"INFO","message":"View with id my_collection_raw_latest already exists in dataset firestore_export."}
Error initializing BigQuery resources: Error initializing latest view: Cannot set a view definition for xxx:firestore_export.my_collection_raw_latest because it is not of type View.
Error importing Collection to BigQuery: Error: Error initializing latest view: Cannot set a view definition for xxx:firestore_export.my_collection_raw_latest because it is not of type View.
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.