apache / apache/datafusion

String compare for Substrait roundtrip check

Open
#8,361 6 comments 0 reactions 0 assignees View on GitHub
bug substrait
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

[roundtrip_with_ctx](https://github.com/apache/arrow-datafusion/blob/aeb012e78115bf69d9a58407ac24bc20bd9e0bf0/datafusion/substrait/tests/cases/roundtrip_logical_plan.rs#L832) function checks for string equality when comparing plans. However, string compare seems to not give correct results. For e.g Here are two plans which are equivalent but considered different by the function.

Plan 1:
```
LeftSemi Join: data.a = __correlated_sq_1.a
Inner Join: data.a = data2.a
TableScan: data projection=[a, b, c, d, e, f]
TableScan: data2 projection=[a, b, c, d, e, f]
SubqueryAlias: __correlated_sq_1
Projection: data2.a
Filter: data2.f NOT IN ([Utf8("a"), Utf8("b"), Utf8("c"), Utf8("d")])
TableScan: data2 projection=[a, f], partial_filters=[data2.f NOT IN ([Utf8("a"), Utf8("b"), Utf8("c"), Utf8("d")])]
```

Plan 2:
```
LeftSemi Join: data.a = data2.a
Inner Join: data.a = data2.a
TableScan: data projection=[a, b, c, d, e, f]
TableScan: data2 projection=[a, b, c, d, e, f]
Projection: data2.a
Filter: data2.f NOT IN ([Utf8("a"), Utf8("b"), Utf8("c"), Utf8("d")])
TableScan: data2 projection=[a, f], partial_filters=[data2.f NOT IN ([Utf8("a"), Utf8("b"), Utf8("c"), Utf8("d")])]
```

It might also give incorrect comparison result in case where there are more than one `partial_filters` but in a different ordering inside the vector.

### To Reproduce

Here is an example testcase I used to produce the plans in the `Describe the bug` section.
```rust
#[tokio::test]
async fn roundtrip_inlist_5() -> Result<()> {
roundtrip("SELECT * FROM data, data2 WHERE data.a = data2.a AND data.a IN (SELECT data2.a FROM data2 WHERE f NOT IN ('a', 'b', 'c', 'd'))").await
}
```

### Expected behavior

The two plans should be considered equivalent.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/substrait/tests/cases/roundtrip_logical_plan.rs at roundtrip_with_ctx and inspect the roundtrip_inlist_5 testcase and its plan comparison. Verify the supplied equivalent plans and cases with multiple partial_filters compare consistently. Done means the roundtrip test accepts the equivalent plans without treating ordering or alias differences as unequal.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.