substrait generated by Apache Calcite does not run in DataFusion
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
This is a report from @lmwnshn
As part of a series of issues that were discovered at CMU while working on DataFusion
- https://github.com/apache/datafusion/issues/14373
Basically they found that DataFusion couldn't run substrait generated by Apache Calcite
Quoting the readme: https://github.com/lmwnshn/15799-s25-project1-remnants/blob/main/README.md
The idea behind this project was to:
1. Convert SQL to a Substrait plan with Calcite.
2. Execute the Substrait plan on different systems (e.g., DuckDB, DataFusion).
Students would then experiment with Calcite's different rules. However, this project idea didn't quite work for the following reasons:
1. substrait-java does not run optimizations - while in theory this can be enabled, I recall running into some issues.
2. More critically, at the time of writing, both DuckDB and DataFusion have limited support for executing the Substrait generated by Calcite.
### To Reproduce
See the code at https://github.com/lmwnshn/15799-s25-project1-remnants
Note I haven't looked at any of this myself so I won't be able to help with specific questionds
### Expected behavior
datafusion should work for all the plans
### Additional context
_No response_
Contributor guide
Research direction
Start with the linked reproduction repository and its README, then review the related issue #14373 and run the Calcite-to-Substrait examples against DataFusion. Identify which generated plans fail and document or address the unsupported cases; done means the reproduced plans execute successfully in DataFusion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100