apache / apache/datafusion

Make `Accumulators` and `ScalarValue` serializable

Open
#11,369 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

When running continuous computations, we'd like to snapshot the state of our operators many of which use accumulators. This would be key to make computations over continuous streams a first class citizen in DataFusion (see #11365 ).

### Describe the solution you'd like

Addition of a `SerializableAccumulator` trait -

```
pub trait SerializableAccumulator: Accumulator {
fn serialize(&self) -> Result>;
fn deserialize(bytes: &[u8]) -> Result>
where
Self: Sized;
}
```

as well as a method on the `Accumulator` trait -

```
fn as_serializable(&self) -> Option<&dyn SerializableAccumulator> {
None
}
```

This would mean `ScalarValue` also needs to implement serialization to [u8]. We have a [POC PR on our fork of DataFusion](https://github.com/probably-nothing-labs/arrow-datafusion/pull/22/files) for this.

Would love to hear feedback from the community on this proposal.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the Accumulator and ScalarValue entry points referenced in this issue, then compare the linked fork POC and read #11365 for the continuous-computation context. Before implementation, resolve the serialization contract with maintainers; done is an agreed scope for both accumulator state and ScalarValue.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.