apache / apache/datafusion

Unify schema usage in Datafusion

Open
#10,442 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Sometimes DF uses 2 schema sources, which is schema from previous node and current like in example(s) below.

Would be nice to check if we can use only 1 schema, multiple schemas are usually confusing

I agree it doesn't make sense -- however, it is the same logic as in `with_new_exprs`:

https://github.com/apache/datafusion/blob/fad16e74229decc40bccf22947d3659cd29fee6f/datafusion/expr/src/logical_plan/plan.rs#L913-L925

_Originally posted by @alamb in https://github.com/apache/datafusion/pull/10405#discussion_r1594332291_

Contributor guide

Open the contributing guide

Research direction

Start with datafusion/expr/src/logical_plan/plan.rs at the referenced lines and compare the schema handling with `with_new_exprs`. Trace the two schema sources described in the issue and determine whether a single schema can be used consistently. Done means the schema usage is unified without changing the intended logical-plan behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.