apache / apache/datafusion

Substrait: Support expression serialization

Open
#7,687 0 comments 2 reactions 0 assignees View on GitHub
enhancement substrait
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

The goal is to allow expressions (not plans and, to start with, scalar expressions) to be passed between different libraries (e.g. pyarrow and datafusion). As a concrete use case we have users in lance who pass in pyarrow.compute filter expressions and we want to use datafusion to satisfy the filter. Currently we do this by going from pyarrow expression to sql-like string and then use the SQL parser but this doesn't work in all cases and substrait is a more natural fit.

### Describe the solution you'd like

About 8 months ago Substrait added support for a new top-level message "ExtendedExpression". This consists of a collection of named expressions and an input schema. Pyarrow is adding support for this in the next release. I've prototyped a solution using datafusion that looks something like this:

```
import pyarrow as pa
import pyarrow.compute as pc
import datafusion.substrait

schema = pa.schema([pa.field("x", pa.int32())])
expr = pc.field("x") + 3
expr_bytes = expr.to_substrait(schema)
expr_sub = datafusion.substrait.substrait.serde.deserialize_expr_bytes(expr_bytes)
expr_df = datafusion.substrait.substrait.consumer.from_substrait_expr(expr_sub)

print("Pyarrow")
print(expr)
print("Datafusion")
print(expr_df)

expr_bytes = datafusion.substrait.substrait.serde.serialize_dfexpr_bytes(expr_df, schema)
expr = pc.Expression.from_substrait(expr_bytes)

print("Pyarrow (return)")
print(expr)
```

Which outputs:

```
Pyarrow
add_checked(x, 3)
Datafusion
Expr(x + Int32(3))
Pyarrow (return)
add_checked(x, 3)
```

### Describe alternatives you've considered

Within datafusion there is also pyarrow_filter_expression which has a similar goal but requires a pyarrow dependency and is limited to pyarrow. Eventually I'd like for there to be support with other languages and I'd also eventually like to support interop with more libraries that have expressions (e.g. polars, ibis)

### Additional context

I plan on implementing this soon. I have a working prototype but it needs cleaned up and a lot of additional testing. If anyone would like to provide some early feedback I can share:

arrow-datafusion changes: https://github.com/apache/arrow-datafusion/compare/main...westonpace:arrow-datafusion:experiment/substrait-extended-expression?expand=1

arrow-datafusion-python changes: https://github.com/apache/arrow-datafusion-python/compare/main...westonpace:arrow-datafusion-python:experiment/substrait-extended-expr?expand=1

Contributor guide

Open the contributing guide

Research direction

Review the linked arrow-datafusion and arrow-datafusion-python prototype branches first, focusing on datafusion.substrait.substrait.serde and consumer entry points such as deserialize_expr_bytes, from_substrait_expr, and serialize_dfexpr_bytes. Done means scalar expressions can be exchanged through Substrait ExtendedExpression between PyArrow and DataFusion, with the additional testing requested in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
backend-api-design, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.