apache / apache/datafusion

Substrait intersection schemas retain primary-input nullability

Open
#25,042 1 comment 0 reactions 1 assignee Claimed by @namanjain24-sudo View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

The Substrait consumer derives all three supported intersection schemas from the primary input's nullability. A field that the intersection makes required can therefore remain nullable in the logical output schema.

Reproduced on `main` at `1b6dc92a221b41088afa2753fe5526be6d057b7b`.

The [Substrait Set Operation rules](https://github.com/substrait-io/substrait/blob/v0.87.0/site/docs/relations/logical_relations.md#set-operation) define different nullability rules for each operation. With the spec's three input patterns (`R` = required, `N` = nullable), all fields having type `i64`:

```text
primary: R R R R N N N N
secondary 1: R R N N R R N N
secondary 2: R N R N R N R N
```

the consumer returns:

| Operation | Expected | Actual |
| --- | --- | --- |
| INTERSECTION_PRIMARY | R R R R R N N N | R R R R N N N N |
| INTERSECTION_MULTISET | R R R R R R R N | R R R R N N N N |
| INTERSECTION_MULTISET_ALL | R R R R R R R N | R R R R N N N N |

### To reproduce

The plans and the table-registration probe are pinned in the [conformance corpus](https://github.com/alexandrefimov/substrait-conformance-cases/tree/f12ba6103217fbd50b95c407b5dc3e8fae1e05a0). From a DataFusion checkout:

```sh
git clone https://github.com/alexandrefimov/substrait-conformance-cases conformance-cases
git -C conformance-cases checkout f12ba6103217fbd50b95c407b5dc3e8fae1e05a0
mkdir -p conformance-inputs datafusion/substrait/examples
cp conformance-cases/derived-schema/setop_*.json conformance-inputs/
cp conformance-cases/probe/datafusion_corpus_probe.rs datafusion/substrait/examples/corpus_probe.rs
SUBSTRAIT_CORPUS_DIR="$PWD/conformance-inputs" cargo run --locked -p datafusion-substrait --example corpus_probe
```

The probe registers empty named tables with exactly those input schemas and prints the logical schema immediately after `from_substrait_plan`, before optimization or execution. The `setop_intersection_*` lines show the three mismatches. Union and primary-minus controls match their respective spec rules; `MINUS_MULTISET` is explicitly unsupported.

### Expected behavior

For multiset intersections, a field is required if any input requires it. For primary intersection, it is nullable only when it is nullable in the primary input and in at least one secondary input.

`from_set_rel` delegates to `LogicalPlanBuilder::intersect`, which uses a left semi join. Keeping the left field nullability there explains the observed schema. This report concerns the schema of consumed Substrait plans; it does not assert incorrect rows. The multiplicity bugs #12955 and #12956 are separate.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.