apache / apache/pinot

Using JSON_MATCH inside of IN_SUBQUERY

Open
#11,050 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Hello! Unsure if this is a bug or an unsupported feature. We are trying to run an aggregation on top of a filtered list of IDs, which we would like to gather using `JSON_MATCH` rather than `JSON_EXTRACT_SCALAR`.

Here is the query:

```
SELECT annotation_type, annotation, COUNT(*)
FROM annotations
WHERE IN_SUBQUERY(
doc_id,
'
SELECT ID_SET(doc_id)
FROM annotations
WHERE JSON_MATCH(annotation, '' "$.id" = ''d0eb419e-c2c4-4815-8085-7a4ea8874dbf'' '')
'
) = 1
GROUP BY 1, 2
ORDER BY 3 DESC
LIMIT 10;
```

It results in this error:
```
[
{
"errorCode": 200,
"message": "QueryExecutionError:\njava.lang.RuntimeException: Caught exception while executing subquery: \n\t\tSELECT ID_SET(doc_id)\n\t\tFROM annotations\n\t\tWHERE JSON_MATCH(annotation, ' \"$.id\" = 'd0eb419e-c2c4-4815-8085-7a4ea8874dbf' ')\n\t"
}
]
```

The subquery itself works fine:
```
SELECT ID_SET(doc_id)
FROM annotations
WHERE JSON_MATCH(annotation, ' "$.id" = ''d0eb419e-c2c4-4815-8085-7a4ea8874dbf'' ');
```

Thought there might be an issue with escaping single quotes, but if you add another set of escaping single quotes around `''d0eb419e-c2c4-4815-8085-7a4ea8874dbf''`, you get the same error (in fact, the extra single quotes are stripped out in the error message). We thought there might be an issue with the double quotes being escaped, but were unable to create any variation that worked.

We can successfully use `JSON_EXTRACT_SCALAR`, but it takes several seconds to run on O(100k) rows:
```
SELECT annotation_type, annotation, count(*)
FROM annotations
WHERE IN_SUBQUERY(
doc_id,
'
SELECT ID_SET(doc_id)
FROM annotation
WHERE JSON_EXTRACT_SCALAR(annotation, ''$.id'', ''STRING'', ''null'') = ''d0eb419e-c2c4-4815-8085-7a4ea8874dbf''
'
) = 1
GROUP BY 1, 2
ORDER BY 3 DESC
LIMIT 10;
```

We'd like to utilize `JSON_MATCH` as it runs faster on the subquery than `JSON_EXTRACT_SCALAR`.

Are we doing something wrong, is this a bug, or is this an unsupported feature? If the last option, is there any plan to add support?

Thank you!

P.S. We are running v0.13.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the v0.13 queries using JSON_MATCH inside IN_SUBQUERY and compare them with the working JSON_EXTRACT_SCALAR form. Trace the subquery execution and quoting behavior to determine whether nested JSON_MATCH is supported; done means the query works correctly or the limitation is documented with a clear explanation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.