Remove `recompute_schema` usage from optimizer
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
The basic assumption that for a given operator we can recompute its schema from inputs' schema is unsound.
- metadata: for plans constructed from SQL metadata will usually be empty, but an application can attach additional metadata to schema or field. The metadata can be assigned on the relational operator (its schema or one of the fields) and may not be derivable from inputs.
- for examples of metadata ussage see https://github.com/apache/datafusion/issues/14247, https://github.com/apache/datafusion/issues/12644, but also other, non-type related use-cases, like primary ID tracking
- field qualification: a plan node may have field qualification retained from inputs or erased, or reassigned. At the optimizer time, we cannot simply assume one way or the other.
- DataFusion deals with plans created by it's own frontend, but DataFusion is also a library. It also deals with plans constructed by other frontends (https://github.com/apache/datafusion/issues/12723). Optimizers need to take any valid plan and produce a valid plan.
The usage of `recompute_schema` within optimizer should be replaced with explicit node schema updates.
For example, when pruning inputs with `RequiredIndices`, the node's schema should be pruned the same way, not recomputed anew.
The usage of `recompute_schema` within analyzer is left for a different issue.
Contributor guide
Research direction
Search the optimizer code for uses of `recompute_schema`, excluding the analyzer work explicitly deferred here, and inspect the `RequiredIndices` handling mentioned in the issue. Replace each optimizer use with explicit schema updates that preserve node metadata and field qualification, then run the relevant optimizer tests to confirm valid schemas after input pruning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100