String compare for Substrait roundtrip check
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
[roundtrip_with_ctx](https://github.com/apache/arrow-datafusion/blob/aeb012e78115bf69d9a58407ac24bc20bd9e0bf0/datafusion/substrait/tests/cases/roundtrip_logical_plan.rs#L832) function checks for string equality when comparing plans. However, string compare seems to not give correct results. For e.g Here are two plans which are equivalent but considered different by the function.
Plan 1:
```
LeftSemi Join: data.a = __correlated_sq_1.a
Inner Join: data.a = data2.a
TableScan: data projection=[a, b, c, d, e, f]
TableScan: data2 projection=[a, b, c, d, e, f]
SubqueryAlias: __correlated_sq_1
Projection: data2.a
Filter: data2.f NOT IN ([Utf8("a"), Utf8("b"), Utf8("c"), Utf8("d")])
TableScan: data2 projection=[a, f], partial_filters=[data2.f NOT IN ([Utf8("a"), Utf8("b"), Utf8("c"), Utf8("d")])]
```
Plan 2:
```
LeftSemi Join: data.a = data2.a
Inner Join: data.a = data2.a
TableScan: data projection=[a, b, c, d, e, f]
TableScan: data2 projection=[a, b, c, d, e, f]
Projection: data2.a
Filter: data2.f NOT IN ([Utf8("a"), Utf8("b"), Utf8("c"), Utf8("d")])
TableScan: data2 projection=[a, f], partial_filters=[data2.f NOT IN ([Utf8("a"), Utf8("b"), Utf8("c"), Utf8("d")])]
```
It might also give incorrect comparison result in case where there are more than one `partial_filters` but in a different ordering inside the vector.
### To Reproduce
Here is an example testcase I used to produce the plans in the `Describe the bug` section.
```rust
#[tokio::test]
async fn roundtrip_inlist_5() -> Result<()> {
roundtrip("SELECT * FROM data, data2 WHERE data.a = data2.a AND data.a IN (SELECT data2.a FROM data2 WHERE f NOT IN ('a', 'b', 'c', 'd'))").await
}
```
### Expected behavior
The two plans should be considered equivalent.
### Additional context
_No response_
Contributor guide
Research direction
Start in datafusion/substrait/tests/cases/roundtrip_logical_plan.rs at roundtrip_with_ctx and inspect the roundtrip_inlist_5 testcase and its plan comparison. Verify the supplied equivalent plans and cases with multiple partial_filters compare consistently. Done means the roundtrip test accepts the equivalent plans without treating ordering or alias differences as unequal.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100