apache / apache/datafusion

make HashTableLookupExpr serializable

Open
#24,277 4 comments 0 reactions 1 assignee Claimed by @barbarj View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

See https://github.com/apache/datafusion/blob/5d6146ca3fc0c0c601c11bd04ed55e6eb4327dfe/datafusion/proto/tests/cases/roundtrip_physical_plan.rs?plain=1#L3963-L3969

If you take an `ExecutionPlan` and serialize it, all occurrences of `HashTableLookupExpr` are replaced with `lit(true)`. This makes it difficult to send dynamic filters across the network.

### Describe the solution you'd like

Ideally, there's some way to serialize and deserialize these expressions, potentially by serializing the seed
```rust
message HashTableLookupExpr {
repeated PhysicalExprNode on_columns = 1;
uint64 seed = 2;
repeated fixed64 build_hashes = 3;
}
```

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.