matrixorigin / matrixorigin/matrixone
[Feature Request]: A new json word breaker.
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
### Is there an existing issue for the same feature request?
- [x] I have checked the existing issues.
### Is your feature request related to a problem?
```Markdown
Use fulltext + json wordbreaker to index json
```
### Describe the feature you'd like
Use fulltext and wordbreaker as json index.
### Describe implementation you've considered
Also want to talk a little bit about how json and fulltext is related at this moment, and what can be done better. Right now, we have a json wordbreaker. It essentially the following json {"a": {"b": "XXX", "c": "YYY"}} into the following keyword, "b=XXX", "c=YYY". Note that only the last step of the path is in the keyword.
We can use this index, select * from T where json_extract_string(jsoncol, "$.a.b") = 'XXX' this can be rewritten as select * from T where json_extract_string(jsoncol, "$.a.b") AND ft_contains(jsoncol, "b=XXX")
Note that we added an AND ft_contians. These ft_contains can be evaluated by fulltext index. But the result is correct, as we will reevalutate json_extract_string. These should all happen within optimizer.
Now, there are different ways to do wordbreaking. First, I should have used our "tuple" encoding, that is, encode the key as tuples ('b', 'XXX'), ('c', 'YYY'), instead of "b=XXX". Reason, we can answer json_extract_float64(....) > 3.14159 Because tuple encoding will encode numerical values into byte string order.
Also, maybe we should encode the tuple as (b, XXX, a), Or if the full path is a.b.c.d....z, we want to encode a key as (z, 'value', a.b.c.....y) Why? you can answer a.b.*.z = value, etc. This is very flexible and there are many other different tricks to encode paths. Make the word breaker configurable (with/without_full_path)
### Documentation, Adoption, Use Case, Migration Strategy
```Markdown
```
### Additional information
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.