apache / apache/datafusion

Use SimpleExtensions for Substrait type variations

Open
#11,544 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Substrait has an extension mechanism for defining things not included in the protobuf format. See https://substrait.io/extensions/#simple-extensions for definition and https://substrait.io/tutorial/sql_to_substrait/#plans for an example (the example only has scalar functions but idea is the same for types and type variations).

DF has been using the extensions for functions, and #11510 adds support for extension types. However DF still uses self-defined consts for type variations (see https://github.com/apache/datafusion/blob/12d82c427d6c37f7884a508707ccd3058a446908/datafusion/substrait/src/variation_const.rs), rather than writing the variations as extensions in the plan. That makes it hard for other producers/consumers to work with DF variations, since they'd need to match the constants, rather than matching by name (+uri).

We should move type variations to also produce simple extensions.

### To Reproduce

_No response_

### Expected behavior

No hardcoded references; type variations written as SimpleExtensionDeclarations à la https://github.com/substrait-io/substrait/blob/a68c1ac62f92d703da624cb8ac0cef854dd2b35f/extensions/type_variations.yaml

### Additional context

DF also doesn't use the extensionUris mechanism correctly, the field isn't filled and all extensionUriReference are just hardcoded into max u32. That's filed separately as https://github.com/apache/datafusion/issues/11545

Contributor guide

Open the contributing guide

Research direction

Start with datafusion/substrait/src/variation_const.rs and compare its type-variation constants with the Substrait SimpleExtensionDeclarations documentation and the type_variations.yaml example. Trace the existing extension-type support referenced by #11510. Done means type variations are emitted as named extensions in plans, with no hardcoded variation references.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.