apache / apache/datafusion

Enable TableScan to return multiple arbitrary table references

Open
#14,358 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

`TableProvider` has a type which indicates which type of table it contains. This can be a view. Therefore it makes sense to encode materialised views using `TableProvider` and use `TableScan`.

Currently `TableScan` does force the use of one or no `TableReference`. Since a view can be derived from multiple tables having the same column names it can happen that two columns can only be distinguished by their respective `TableReference`.

### Describe the solution you'd like

`TableScan` should contain unique `table_name: TableReference` for each column.

### Describe alternatives you've considered

Even through `TableScan` contains a `DFSchema` which contains a list of `TableReference`, it is not possible to use it since some optimisation rules recreate the `TableScan` with the wrong schema.

1. We change the schema in `TableProvider` to `DFSchema` instead of `Schema`. I did not test if having multiple same columns in `TableProvider` does cause issues which would force this option. As in the execution only the column position is important, this seems unlikely.
2. In my current implementation I am inlining table names using `.flat_name()` and outline them again using a analyser rule. Since I can't use `TableScan` I am using a custom Extension which is resolved by the analyser too.
3. Leaving this to the user or creating some new alternative `LogicalPlan` node is also possible.

### Additional context

This feature is slightly related to #14310.

Contributor guide

Open the contributing guide

Research direction

Start by reading the TableScan and TableProvider definitions, then inspect the optimizer rules that recreate TableScan with schemas. Review the relationship to issue #14310 and compare the listed alternatives. Done means TableScan can preserve a unique TableReference for every column without breaking schema handling or optimization.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.