apache / apache/pinot

Support FLATTEN (unnest), KVGEN type functions

Open
#7,881 3 comments 6 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

We current flatten (virtually) the JSON records during JSON index creation. Also, there was a feature recently added to flatten the JSON data (at any chosen subtree level) during ingestion which physically stores flattened records / schema.

We should consider exposing FLATTEN function in SQL for flexible exploration of repeated data in JSON

Take the following example

```
{
"name": "abc",
"age": 25,
"phones": [123, 456, 789]
}
```

Doing something like `SELECT FLATTEN(person) FROM FOO` should result in following three records

```
abc 25 123
abc 25 456
abc 25 789
```

Once FLATTEN has been used to unnest arrays, we can then do deeper analysis on resulting transient records. Take the following snippet of a single record from [yelp business dataset](https://www.yelp.com/dataset/documentation/main)

```
{
"name": "Garaje",
// an array of strings of business categories
"categories": [
"Mexican",
"Burgers",
"Gastropubs"
]
}
```

Upon flattening the categories array, we can apply group by and aggregation. Something like following ...

Note - not implying query syntax. Just writing something for the sake of idea/explanation

```
SELECT inner:flattenedCategories, count(inner:flattenedCategories)
FROM
(SELECT FLATTEN(categories) flattenedCategories FROM foo) inner
GROUP BY inner:flattenedCategories
ORDER BY count(inner:flattenedCategories)
```

```
Mexican 100
Burgers 50
GastroPubs 500
```

In addition to FLATTEN, KVGEN or something similar can be another useful function especially for arbitrary maps where both key and value can represent data instead of a schema member / field. Essentially the map can contain unknown element names

Consider the following 2 records

```
{
"sales": {
"2021-12-10": 100000,
"2021-12-11": 50000
}
}
{
"sales": {
"2021-12-10": 20000,
"2021-12-15": 80000
}
}
```

Simply applying KVGEN as `SELECT KVGEN(col) FROM FOO` can result in following 2 records

```
[{"key":"2021-12-10","value":"100000"},{"key":"2021-12-11","value":"50000"}]
[{"key":"2021-12-10","value":"20000"},{"key":"2021-12-15","value":"80000"}]
```

Now combine this with FLATTEN

SELECT FLATTEN(KVGEN(col)) FROM FOO

```
{"key":"2021-12-10","value":"100000"}
{"key":"2021-12-11","value":"50000"}
{"key":"2021-12-10","value":"20000"}
{"key":"2021-12-15","value":"80000"}
```

Now that we have key-value separated and flattened, we can do further analysis

```
SELECT SUM(inner:flattenedSales.`value`) AS totalSales
FROM (SELECT FLATTEN(KVGEN(sales)) flattenedSales FROM FOO) inner;
```

`250000`

```
SELECT
inner:flattenedSales.`key` AS Date,
SUM(inner:flattenedSales.`value`) AS TotalSales
FROM (SELECT FLATTEN(KVGEN(sales)) flattenedSales FROM FOO) inner
GROUP BY inner:flattenedSales.`key`
```

```
Date. TotalSales
2021-12-10 120000
2021-12-11 50000
2021-12-15 80000
```

Contributor guide

Open the contributing guide

Research direction

The issue names no implementation files, tests, or entry points. Start by reviewing Pinot's SQL function and JSON query support, then define the FLATTEN and KVGEN syntax, semantics, type behavior, and query-planning requirements; done means an agreed design with coverage for the array and map examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.