kernelci / kernelci/kcidb

bigquery: Consider flattening data structures

Open
#212 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
26
Forks
40
PR merge metrics
No merged PRs in 30d

Description

Right now we're using `RECORD` types in our BigQuery schema, and we also use arrays. However, that doesn't work well with all BigQuery features. For example, materialized views don't support structure (`RECORD`) arguments in ANY_VALUE aggregations, and arrays, which don't support equality operations, don't work with `UNION DISTINCT`.

Consider flattening our data structures into JSON fields in BigQuery, similarly to what we do in SQLite. We can still extract the data from them server-side with JSON functions (although not so efficiently).

Forked off #183.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the BigQuery schema definitions and comparing them with the SQLite JSON representation mentioned in the issue. No files or tests are named, so first identify the affected schema and query paths. Done means deciding and implementing how RECORD and array data should be represented as JSON while preserving server-side extraction.

Written by the indexing model from the issue text.

Assessment

Tech stack
json
Domain
databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.