[Discuss] Backwards compatibility for `datafusion-proto`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
https://docs.rs/datafusion-proto/latest/datafusion_proto/#version-compatibility says:
> The serialized form are not guaranteed to be compatible across DataFusion versions. A plan serialized with one version of DataFusion may not be able to deserialized with a different version.
However, as these messages become more widely used (e.g. datafusion-proto is now used to send `Expr`s over the FFI boundary, as mentioned by @timesaucer) so having them be incompatible may cause bugs / unintended side effects
Also, we have some recent contributions such as the following from @tobixdev and @paleolimbot that actually do attempt to maintain backwards compatibility
- https://github.com/apache/datafusion/pull/17986/files#r2442433612
Given the protobuf definitions aren't changing all that frequently anymore I think it may be worth reconsidering our stance on compatibility
### Describe the solution you'd like
1. Consider if we should make a stronger statement about backwards compatibility of protobuf serialized Exprs
### Describe alternatives you've considered
Maybe we can use substrait for this?
If we do choose to make a stronger statement, we should also consider how we can test / verify it (aka perhaps check in stored protobuf exprs and ensure we can continue to serialize them in subsequent releases)
### Additional context
_No response_
Contributor guide
Research direction
Start with the datafusion-proto version-compatibility statement in the linked docs and inspect the referenced PR changes. Determine whether a compatibility guarantee is intended for serialized protobuf Exprs across releases, then define stored-message or equivalent verification coverage; done means an agreed compatibility policy and tests that enforce it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100