apache / apache/datafusion

Make EquivalenceProperties::new_with_schema not require aligned schema

Open
#18,933 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

There generally has requirements for renaming, appending, removing a column from the input plan.
If the input plan has an equivalence properties, it should be preserved as much as possible when the schema is altered.

However, the current `EquivalenceProperties::new_with_schema` implementation requires the new schema and the original schema being aligned. Specifically, the number of columns and column positions should be consistent. This constraint disallows us to consistently update the equivalence properties when the schema is altered.

I wonder if we can remove the align constraint.

Contributor guide

Open the contributing guide

Research direction

Locate the implementation and callers of EquivalenceProperties::new_with_schema, then trace how equivalence properties are used when input columns are renamed, appended, or removed. Determine how altered schemas can preserve the valid properties without requiring aligned column counts or positions. Done means the alignment constraint is removed while existing properties remain correct for these schema changes, with regression coverage for the supported cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.