bigquery: Consider flattening data structures
- Dominant language
- Python
- Stars
- 26
- Forks
- 40
- PR merge metrics
- No merged PRs in 30d
Description
Right now we're using `RECORD` types in our BigQuery schema, and we also use arrays. However, that doesn't work well with all BigQuery features. For example, materialized views don't support structure (`RECORD`) arguments in ANY_VALUE aggregations, and arrays, which don't support equality operations, don't work with `UNION DISTINCT`.
Consider flattening our data structures into JSON fields in BigQuery, similarly to what we do in SQLite. We can still extract the data from them server-side with JSON functions (although not so efficiently).
Forked off #183.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the BigQuery schema definitions and comparing them with the SQLite JSON representation mentioned in the issue. No files or tests are named, so first identify the affected schema and query paths. Done means deciding and implementing how RECORD and array data should be represented as JSON while preserving server-side extraction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- json
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100