firebase / firebase/extensions
🐛 firestore-bigquery-export Using the fs-bq-schema-views script for nested arrays of maps
- Dominant language
- TypeScript
- Stars
- 979
- Forks
- 433
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 125
Description
This feature request is for extension: `firestore-bigquery-export`, when using the `fs-bq-schema-views` script
### What feature would you like to see?
When using the Firestore to BigQuery extension or the `firestore-bigquery-export` command, two types of tables are generated, prefix_changelog and prefix_raw, that cannot be used as such. In order to create views, the script `fs-bq-schema-views` uses user defined json schemas to create views from raw tables. All is explained [here](https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/guides/GENERATE_SCHEMA_VIEWS.md).
However, raw data can be pretty complex and the script doesn't cover all cases. In particular, nested objects, like arrays made of maps. [This issue](https://github.com/firebase/extensions/issues/573) also discussed it, but the problem was not solved.
You can check the related merge request [here](https://github.com/firebase/extensions/pull/1366).
I have tested the format with the use of array_map, see [user_complex.json](https://github.com/firebase/extensions/pull/1366/files#diff-9109260abe72f2f2b9646f60a1b3fdd95748f41373d40b3c843d0d8c4abb539a), which did not work at all, and the [complexSchema.json](https://github.com/firebase/extensions/pull/1366/files#diff-9936fe6eb699b405a305f3c8547bb7e243ee35fd71518be719279784358fd650), but it only resulted in the json object.
### How would you use it?
The feature supports typed arrays, but it does not seem to work for arrays of maps, like in this example:
```javascript
{
"fields": [
{
"name": "nickname",
"type": "string"
},
{
"name": "location",
"type": "geopoint"
},
{
"name": "profile",
"type": "map",
"fields": [
{
"name": "birthDate",
"type": "timestamp"
},
{
"name": "hand",
"type": "string"
},
{
"name": "levels",
"type": "array",
"fields": [
{
"name": "date",
"type": "timestamp",
"column_name": "levelDate"
},
{
"name": "value",
"type": "number",
"column_name": "levelValue"
}
]
}
]
}
]
}
```
Contributor guide
Research direction
Start with the fs-bq-schema-views script and GENERATE_SCHEMA_VIEWS.md, then compare the array_map example in user_complex.json with complexSchema.json and the related pull request. Done means the schema format generates usable views with typed columns for nested arrays of maps, rather than only a JSON object.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100