apache / apache/datafusion

Extending BinaryOp/Operator support

Open
#15,326 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

There are some gaps between our [`Operator`](https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.Operator.html) and [`BinaryOperator`](https://docs.rs/sqlparser/0.55.0/sqlparser/ast/enum.BinaryOperator.html) from sqlparser, where some of them are useful.

This ticket is not intended to align the gap between `Operator` and `BinaryOperator`. But to include those might-helpful operators with sqlparser as the reference. And further implement some of them (which means leaving another part just a placeholder, and even not included).

We can categorize our current `Operator`s into three parts: fully supported (like AND/OR), included but not supported (like `@>`) and not included. This ticket focuses on extending the first two parts. And for some operators, it may take a relatively long time to fully support them. Like those for JSON operation. The meaning of listing them here is for rough progress tracking, and to push them into the "included but not implemented" stage for downstream projects to easily extend their version.

### Describe the solution you'd like

Here is a table of various operators and their status:

| Operator | Description | Status | Tickets |
|:----------:|:----------|---|---|
| [Arrow](https://docs.rs/sqlparser/0.55.0/sqlparser/ast/enum.BinaryOperator.html#variant.Arrow) | (PG) `->` extracts a JSON object field or array element | Included But Not Implemented | #15327, |
| [LongArrow](https://docs.rs/sqlparser/0.55.0/sqlparser/ast/enum.BinaryOperator.html#variant.LongArrow) | (PG) `->>` extracts a JSON object field or JSON array element and converts it to text | Included But Not Implemented | #15327, |
| [HashArrow](https://docs.rs/sqlparser/0.55.0/sqlparser/ast/enum.BinaryOperator.html#variant.HashArrow) | (PG) `#>` extracts a JSON sub-object at the specified path | Included But Not Implemented | #15327, |
| [HashLongArrow](https://docs.rs/sqlparser/0.55.0/sqlparser/ast/enum.BinaryOperator.html#variant.HashLongArrow) | (PG) `#>>` extracts JSON sub-object at the specified path | Included But Not Implemented | #15327, |
| [ArrowAt](https://docs.rs/sqlparser/0.55.0/sqlparser/ast/enum.BinaryOperator.html#variant.ArrowAt) | (PG) `<@` is used for JSON and text searches | 🚧 Available for array, not implemented for structs and text | #6885, #15327, |
| [AtArrow](https://docs.rs/sqlparser/0.55.0/sqlparser/ast/enum.BinaryOperator.html#variant.AtArrow) | (PG) `@>` is used for JSON and text searches | 🚧 Available for array, not implemented for structs and text | #6885, #15327, |
| [AtAt](https://docs.rs/sqlparser/0.55.0/sqlparser/ast/enum.BinaryOperator.html#variant.AtAt) | (PG) `@@` is used for JSON and text searches | Included But Not Implemented | #15327, |
| [XOR](https://docs.rs/sqlparser/0.55.0/sqlparser/ast/enum.BinaryOperator.html#variant.XOR) | `a XOR b`. We've implemented `BitwiseXor` and can map this to `BitwiseXor` operator | ✅ map to `BitwiseXor` | #15327 |
| [`DIV`](https://docs.rs/sqlparser/0.55.0/sqlparser/ast/enum.BinaryOperator.html#variant.MyIntegerDivide) or [`//`](https://docs.rs/sqlparser/0.55.0/sqlparser/ast/enum.BinaryOperator.html#variant.DuckIntegerDivide) | (MySQL/DuckDB) Integer division. | Not Included | |
| `ALL` | `SELECT 1 = ALL(ARRAY[1,1,1,1]);` | Unparsed | |
| [Question](https://docs.rs/sqlparser/0.55.0/sqlparser/ast/enum.BinaryOperator.html#variant.Question) | (PG) `?` checks whether a string exists as a top-level key within the JSON value | Included But Not Implemented | #15327, |
| [AtQuestion](https://docs.rs/sqlparser/0.55.0/sqlparser/ast/enum.BinaryOperator.html#variant.AtQuestion) | (PG) `@?` checks the given JSON path returns an item for the JSON value | Included But Not Implemented | #15327, |
| [QuestionAnd](https://docs.rs/sqlparser/0.55.0/sqlparser/ast/enum.BinaryOperator.html#variant.QuestionAnd) | (PG) `?&` check whether all of the the indicated array members exist as top-level keys | Included But Not Implemented | #15327, |
| [QuestionPipe](https://docs.rs/sqlparser/0.55.0/sqlparser/ast/enum.BinaryOperator.html#variant.QuestionPipe) | (PG) `?` checks whether any of the the indicated array members exist as top-level keys | Included But Not Implemented | #15327, |

Feel free to add or modify it

### Describe alternatives you've considered

_No response_

### Additional context

Cross-references:
- https://github.com/apache/datafusion/issues/7308
- https://github.com/apache/datafusion/issues/14660
- https://github.com/apache/datafusion/issues/2547

Contributor guide

Open the contributing guide

Research direction

Start by comparing DataFusion's Operator documentation with sqlparser's BinaryOperator documentation, using the operator table as the scope guide. Review the linked follow-up tickets, especially #15327, before choosing a single operator or group. Done means the selected operators have an agreed support status and the corresponding implementation or tracking work is complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.