apache / apache/datafusion

[Discuss] Backwards compatibility for `datafusion-proto`

Open
#18,270 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

https://docs.rs/datafusion-proto/latest/datafusion_proto/#version-compatibility says:

> The serialized form are not guaranteed to be compatible across DataFusion versions. A plan serialized with one version of DataFusion may not be able to deserialized with a different version.

However, as these messages become more widely used (e.g. datafusion-proto is now used to send `Expr`s over the FFI boundary, as mentioned by @timesaucer) so having them be incompatible may cause bugs / unintended side effects

Also, we have some recent contributions such as the following from @tobixdev and @paleolimbot that actually do attempt to maintain backwards compatibility
- https://github.com/apache/datafusion/pull/17986/files#r2442433612

Given the protobuf definitions aren't changing all that frequently anymore I think it may be worth reconsidering our stance on compatibility

### Describe the solution you'd like

1. Consider if we should make a stronger statement about backwards compatibility of protobuf serialized Exprs

### Describe alternatives you've considered

Maybe we can use substrait for this?

If we do choose to make a stronger statement, we should also consider how we can test / verify it (aka perhaps check in stored protobuf exprs and ensure we can continue to serialize them in subsequent releases)

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the datafusion-proto version-compatibility statement in the linked docs and inspect the referenced PR changes. Determine whether a compatibility guarantee is intended for serialized protobuf Exprs across releases, then define stored-message or equivalent verification coverage; done means an agreed compatibility policy and tests that enforce it.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.