apache / apache/datafusion

substrait generated by Apache Calcite does not run in DataFusion

Open
#14,831 7 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

This is a report from @lmwnshn

As part of a series of issues that were discovered at CMU while working on DataFusion
- https://github.com/apache/datafusion/issues/14373

Basically they found that DataFusion couldn't run substrait generated by Apache Calcite

Quoting the readme: https://github.com/lmwnshn/15799-s25-project1-remnants/blob/main/README.md

The idea behind this project was to:

1. Convert SQL to a Substrait plan with Calcite.
2. Execute the Substrait plan on different systems (e.g., DuckDB, DataFusion).
Students would then experiment with Calcite's different rules. However, this project idea didn't quite work for the following reasons:

1. substrait-java does not run optimizations - while in theory this can be enabled, I recall running into some issues.
2. More critically, at the time of writing, both DuckDB and DataFusion have limited support for executing the Substrait generated by Calcite.

### To Reproduce

See the code at https://github.com/lmwnshn/15799-s25-project1-remnants

Note I haven't looked at any of this myself so I won't be able to help with specific questionds

### Expected behavior

datafusion should work for all the plans

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the linked reproduction repository and its README, then review the related issue #14373 and run the Calcite-to-Substrait examples against DataFusion. Identify which generated plans fail and document or address the unsupported cases; done means the reproduced plans execute successfully in DataFusion.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.