Enable TableScan to return multiple arbitrary table references
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
`TableProvider` has a type which indicates which type of table it contains. This can be a view. Therefore it makes sense to encode materialised views using `TableProvider` and use `TableScan`.
Currently `TableScan` does force the use of one or no `TableReference`. Since a view can be derived from multiple tables having the same column names it can happen that two columns can only be distinguished by their respective `TableReference`.
### Describe the solution you'd like
`TableScan` should contain unique `table_name: TableReference` for each column.
### Describe alternatives you've considered
Even through `TableScan` contains a `DFSchema` which contains a list of `TableReference`, it is not possible to use it since some optimisation rules recreate the `TableScan` with the wrong schema.
1. We change the schema in `TableProvider` to `DFSchema` instead of `Schema`. I did not test if having multiple same columns in `TableProvider` does cause issues which would force this option. As in the execution only the column position is important, this seems unlikely.
2. In my current implementation I am inlining table names using `.flat_name()` and outline them again using a analyser rule. Since I can't use `TableScan` I am using a custom Extension which is resolved by the analyser too.
3. Leaving this to the user or creating some new alternative `LogicalPlan` node is also possible.
### Additional context
This feature is slightly related to #14310.
Contributor guide
Research direction
Start by reading the TableScan and TableProvider definitions, then inspect the optimizer rules that recreate TableScan with schemas. Review the relationship to issue #14310 and compare the listed alternatives. Done means TableScan can preserve a unique TableReference for every column without breaking schema handling or optimization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100