Unify schema usage in Datafusion
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
Sometimes DF uses 2 schema sources, which is schema from previous node and current like in example(s) below.
Would be nice to check if we can use only 1 schema, multiple schemas are usually confusing
I agree it doesn't make sense -- however, it is the same logic as in `with_new_exprs`:
https://github.com/apache/datafusion/blob/fad16e74229decc40bccf22947d3659cd29fee6f/datafusion/expr/src/logical_plan/plan.rs#L913-L925
_Originally posted by @alamb in https://github.com/apache/datafusion/pull/10405#discussion_r1594332291_
Contributor guide
Research direction
Start with datafusion/expr/src/logical_plan/plan.rs at the referenced lines and compare the schema handling with `with_new_exprs`. Trace the two schema sources described in the issue and determine whether a single schema can be used consistently. Done means the schema usage is unified without changing the intended logical-plan behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100