Make `Accumulators` and `ScalarValue` serializable
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
When running continuous computations, we'd like to snapshot the state of our operators many of which use accumulators. This would be key to make computations over continuous streams a first class citizen in DataFusion (see #11365 ).
### Describe the solution you'd like
Addition of a `SerializableAccumulator` trait -
```
pub trait SerializableAccumulator: Accumulator {
fn serialize(&self) -> Result>;
fn deserialize(bytes: &[u8]) -> Result>
where
Self: Sized;
}
```
as well as a method on the `Accumulator` trait -
```
fn as_serializable(&self) -> Option<&dyn SerializableAccumulator> {
None
}
```
This would mean `ScalarValue` also needs to implement serialization to [u8]. We have a [POC PR on our fork of DataFusion](https://github.com/probably-nothing-labs/arrow-datafusion/pull/22/files) for this.
Would love to hear feedback from the community on this proposal.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start with the Accumulator and ScalarValue entry points referenced in this issue, then compare the linked fork POC and read #11365 for the continuous-computation context. Before implementation, resolve the serialization contract with maintainers; done is an agreed scope for both accumulator state and ScalarValue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100