json-path filter by path existence broken due to optimization
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 31m
- Merged PRs (30d)
- 209
Description
#6654 introduced a bug that causes filter expressions that are checking for path existence to not function correctly. Using this example from the json-path docs:
```
$..book[?(@.isbn)]
```
it incorrectly will return all 4 books, even though only 2 of them have the 'isbn' property, due to special handling of the `UNDEFINED` placeholder object in this case.
Unfortunately, I cannot find a way to workaround this issue, so we might want to consider reverting that commit, or if we are ok with the loss of this functionality, then we should document this behavior. Also of note, depending on the type of transform that would probably be done to actually use the 'isbn' value during ingestion, it may or may not work correctly. Collecting the list of isbn values, e.g.
```
$..book[?(@.isbn)].isbn
```
say to select a list of isbn numbers to use as a multi-value dimension or whatever, does function correctly with the optimization since `null` values are filtered out, so this might not be a big deal, but I would be willing to suspect there are cases where things will not function as expected, such as selecting 'author' field where 'isbn' is not null, e.g.
```
$..book[?(@.isbn)].author
```
I am uneasy that it broke full behavioral compatibility. Still, the performance difference of the optimization is significant so maybe careful consideration is in order? I would put the chances of getting the performance fixed upstream at very low because of lack of development over the last year (see https://github.com/json-path/JsonPath/issues/500)
Contributor guide
Research direction
Review #6654 and the linked json-path documentation, then reproduce the four-book examples for path-existence filters and value selection. Compare the optimized behavior with the expected results and determine whether compatibility can be restored or the behavior must be documented; done means the chosen resolution is implemented with regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100