Support FLATTEN (unnest), KVGEN type functions
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
We current flatten (virtually) the JSON records during JSON index creation. Also, there was a feature recently added to flatten the JSON data (at any chosen subtree level) during ingestion which physically stores flattened records / schema.
We should consider exposing FLATTEN function in SQL for flexible exploration of repeated data in JSON
Take the following example
```
{
"name": "abc",
"age": 25,
"phones": [123, 456, 789]
}
```
Doing something like `SELECT FLATTEN(person) FROM FOO` should result in following three records
```
abc 25 123
abc 25 456
abc 25 789
```
Once FLATTEN has been used to unnest arrays, we can then do deeper analysis on resulting transient records. Take the following snippet of a single record from [yelp business dataset](https://www.yelp.com/dataset/documentation/main)
```
{
"name": "Garaje",
// an array of strings of business categories
"categories": [
"Mexican",
"Burgers",
"Gastropubs"
]
}
```
Upon flattening the categories array, we can apply group by and aggregation. Something like following ...
Note - not implying query syntax. Just writing something for the sake of idea/explanation
```
SELECT inner:flattenedCategories, count(inner:flattenedCategories)
FROM
(SELECT FLATTEN(categories) flattenedCategories FROM foo) inner
GROUP BY inner:flattenedCategories
ORDER BY count(inner:flattenedCategories)
```
```
Mexican 100
Burgers 50
GastroPubs 500
```
In addition to FLATTEN, KVGEN or something similar can be another useful function especially for arbitrary maps where both key and value can represent data instead of a schema member / field. Essentially the map can contain unknown element names
Consider the following 2 records
```
{
"sales": {
"2021-12-10": 100000,
"2021-12-11": 50000
}
}
{
"sales": {
"2021-12-10": 20000,
"2021-12-15": 80000
}
}
```
Simply applying KVGEN as `SELECT KVGEN(col) FROM FOO` can result in following 2 records
```
[{"key":"2021-12-10","value":"100000"},{"key":"2021-12-11","value":"50000"}]
[{"key":"2021-12-10","value":"20000"},{"key":"2021-12-15","value":"80000"}]
```
Now combine this with FLATTEN
SELECT FLATTEN(KVGEN(col)) FROM FOO
```
{"key":"2021-12-10","value":"100000"}
{"key":"2021-12-11","value":"50000"}
{"key":"2021-12-10","value":"20000"}
{"key":"2021-12-15","value":"80000"}
```
Now that we have key-value separated and flattened, we can do further analysis
```
SELECT SUM(inner:flattenedSales.`value`) AS totalSales
FROM (SELECT FLATTEN(KVGEN(sales)) flattenedSales FROM FOO) inner;
```
`250000`
```
SELECT
inner:flattenedSales.`key` AS Date,
SUM(inner:flattenedSales.`value`) AS TotalSales
FROM (SELECT FLATTEN(KVGEN(sales)) flattenedSales FROM FOO) inner
GROUP BY inner:flattenedSales.`key`
```
```
Date. TotalSales
2021-12-10 120000
2021-12-11 50000
2021-12-15 80000
```
Contributor guide
Research direction
The issue names no implementation files, tests, or entry points. Start by reviewing Pinot's SQL function and JSON query support, then define the FLATTEN and KVGEN syntax, semantics, type behavior, and query-planning requirements; done means an agreed design with coverage for the array and map examples.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100