firebase / firebase/extensions

🐛 firestore-bigquery-export Using the fs-bq-schema-views script for nested arrays of maps

Open
#1,963 3 comments 2 reactions 0 assignees View on GitHub
extension: firestore-bigquery-export type: feature request
Dominant language
TypeScript
Stars
979
Forks
433
Avg merge
1d 13h
Merged PRs (30d)
125

Description

This feature request is for extension: `firestore-bigquery-export`, when using the `fs-bq-schema-views` script

### What feature would you like to see?

When using the Firestore to BigQuery extension or the `firestore-bigquery-export` command, two types of tables are generated, prefix_changelog and prefix_raw, that cannot be used as such. In order to create views, the script `fs-bq-schema-views` uses user defined json schemas to create views from raw tables. All is explained [here](https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/guides/GENERATE_SCHEMA_VIEWS.md).

However, raw data can be pretty complex and the script doesn't cover all cases. In particular, nested objects, like arrays made of maps. [This issue](https://github.com/firebase/extensions/issues/573) also discussed it, but the problem was not solved.
You can check the related merge request [here](https://github.com/firebase/extensions/pull/1366).
I have tested the format with the use of array_map, see [user_complex.json](https://github.com/firebase/extensions/pull/1366/files#diff-9109260abe72f2f2b9646f60a1b3fdd95748f41373d40b3c843d0d8c4abb539a), which did not work at all, and the [complexSchema.json](https://github.com/firebase/extensions/pull/1366/files#diff-9936fe6eb699b405a305f3c8547bb7e243ee35fd71518be719279784358fd650), but it only resulted in the json object.

### How would you use it?

The feature supports typed arrays, but it does not seem to work for arrays of maps, like in this example:

```javascript
{
"fields": [
{
"name": "nickname",
"type": "string"
},
{
"name": "location",
"type": "geopoint"
},
{
"name": "profile",
"type": "map",
"fields": [
{
"name": "birthDate",
"type": "timestamp"
},
{
"name": "hand",
"type": "string"
},
{
"name": "levels",
"type": "array",
"fields": [
{
"name": "date",
"type": "timestamp",
"column_name": "levelDate"
},
{
"name": "value",
"type": "number",
"column_name": "levelValue"
}
]
}
]
}
]
}
```

Contributor guide

Open the contributing guide

Research direction

Start with the fs-bq-schema-views script and GENERATE_SCHEMA_VIEWS.md, then compare the array_map example in user_complex.json with complexSchema.json and the related pull request. Done means the schema format generates usable views with typed columns for nested arrays of maps, rather than only a JSON object.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.