apache / apache/druid

json-path filter by path existence broken due to optimization

Open
#6,680 1 comment 0 reactions 0 assignees View on GitHub
Bug stale
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 31m
Merged PRs (30d)
209

Description

#6654 introduced a bug that causes filter expressions that are checking for path existence to not function correctly. Using this example from the json-path docs:
```
$..book[?(@.isbn)]
```
it incorrectly will return all 4 books, even though only 2 of them have the 'isbn' property, due to special handling of the `UNDEFINED` placeholder object in this case.

Unfortunately, I cannot find a way to workaround this issue, so we might want to consider reverting that commit, or if we are ok with the loss of this functionality, then we should document this behavior. Also of note, depending on the type of transform that would probably be done to actually use the 'isbn' value during ingestion, it may or may not work correctly. Collecting the list of isbn values, e.g.
```
$..book[?(@.isbn)].isbn
```
say to select a list of isbn numbers to use as a multi-value dimension or whatever, does function correctly with the optimization since `null` values are filtered out, so this might not be a big deal, but I would be willing to suspect there are cases where things will not function as expected, such as selecting 'author' field where 'isbn' is not null, e.g.
```
$..book[?(@.isbn)].author
```
I am uneasy that it broke full behavioral compatibility. Still, the performance difference of the optimization is significant so maybe careful consideration is in order? I would put the chances of getting the performance fixed upstream at very low because of lack of development over the last year (see https://github.com/json-path/JsonPath/issues/500)

Contributor guide

Open the contributing guide

Research direction

Review #6654 and the linked json-path documentation, then reproduce the four-book examples for path-existence filters and value selection. Compare the optimized behavior with the expected results and determine whether compatibility can be restored or the behavior must be documented; done means the chosen resolution is implemented with regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.