apache / apache/datafusion

Type variation in substrait: Hardcoding the numbers isn't really the proper way to do type variations.

Open
#12,355 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

> FWIW, hardcoding the numbers isn't really the proper way to do type variations. (Rather we should add the variation as an extension and refer to the extension's id.) However, given this is already used for default vs large, I guess adding view makes sense - and they can all be migrated at once to the proper way someday.

_Originally posted by @Blizzara in https://github.com/apache/datafusion/pull/12199#discussion_r1734657314_

The idea here is for someone to research how this is supposed to work and then change the datafusion substrait implementation to do it

Contributor guide

Open the contributing guide

Research direction

Start by researching how Substrait type variations are represented as extensions, then inspect DataFusion's Substrait implementation for the existing default, large, and view handling. Done means the implementation uses extension IDs rather than hardcoded variation numbers, with all three existing variations accounted for.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.