apache / apache/datafusion

[substrait] Add support for ExtensionTable

Open
#13,771 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Custom TableProvider implementations cannot currently be encoded as ExtensionTables in Substrait. The Substrait plan will only retain the table names, which are hardly ever enough to restore the table definition on the consumer side. For UDTFs in particular, the name is completely useless, as it's always `tmp_table`.

### Describe the solution you'd like

Add two more methods in `SerializerRegistry` for serializing/deserializing `TableSource` instances and use these new extensions in `to_substrait_plan` and `from_substrait_plan` to let users encode/decode custom table definitions as ExtensionTables.

### Describe alternatives you've considered

For the cases where the user controls the table name, one hideous workaround would be to encode the whole table definition in the table name and register a custom schema provider to decode it (e.g. `some_catalog.custom_schema."base64(proto_binary)"`). This is a horrible hack as it requires using those names in SQL queries and it doesn't work for UDTFs.
A far better alternative is to leverage the already supported Substrait extensions (in particular, `ExtensionLeaf`), by implementing the `SerializerRegistry` trait and forcing the table to fit into a `UserDefinedLogicalNode`s.
However, this approach is both limited and unnatural:
- logical plans have to be preprocessed in order to replace `TableScan`s with `Extension` nodes before converting to substrait
- The logical plan resulting from decoding Substrait can only be executed if an `ExtensionPlanner` is registered for handling the user-defined nodes, but in this case it would not benefit from the special treatment that tables get in DataFusion (projections and filter pushdowns for scan, various knobs to instruct the engine about the table capabilities etc.). Rewriting the decoded plan to convert the user-defined node back to a TableScan is the only way to benefit from all that.
- Substrait also encodes projections only for tables (i.e. `ReadRel`s), so an ExtensionLeaf can't make use of that. Rewriting the Substrait plan itself to overcome this limitation is way more tedious than it should be.

### Additional context

This should be a child issue of https://github.com/apache/datafusion/issues/13318.

Contributor guide

Open the contributing guide

Research direction

Start by reading SerializerRegistry and the TableSource serialization paths, then trace how to_substrait_plan and from_substrait_plan currently represent tables. Implement the requested serialization and deserialization extensions for ExtensionTables, including custom table definitions and UDTFs, and verify that plans round-trip with their table information intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.