[Proposal] Event Sourcing support for grain requested snapshots, log truncation and log deletion
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 13h 56m
- Merged PRs (30d)
- 351
Description
Implement state snapshots, log truncation and log deletion functions that event sourced grains can use to better manage their state, log retention and performance.
Snapshots would allow event sourced grains to request their versioned state is stored by the log consistency provider to allow for faster subsequent building of state by reading the snapshot + subsequent events. The grain is best placed to determine when snapshots should happen, based on technical or business requirements (This would address https://github.com/dotnet/orleans/issues/4862 and allow it to be closed)
Log truncation allows event sourced grains to request the log consistency provider discards prior entries in the log, this could be used where an event sourced grain contains an audit history and only needs recent state.
Log deletion is the event sourcing analogy of `IGrainStorage.ClearStateAsync` and would clear the existing log and state for an event sourced grain.
`JournaledGrain` would have some additional protected methods:
```csharp
public abstract class JournaledGrain :
...
{
protected ValueTask SnapshotAsync(bool truncateLog = false, CancellationToken token = default);
protected ValueTask TruncateLogAsync(int beforeVersion, CancellationToken token = default);
protected ValueTask ClearLogAsync(CancellationToken token = default);
protected ValueTask TryClearLogAsync(int version, CancellationToken token = default);
}
```
`ValueTask SnapshotAsync()` which would ask the underlying log consistency provider to snapshot the grain state + version, and return the version on which the snapshot was generated. If the underlying provider is storing the snapshot in the log then it's required to handle race-conditions in the log if there are multiple writers for example (through reentracy or externally), could optionally have a truncate parameter to indicate whether to also truncate the log after snapshotting - asking the provider to ensure atomicity and avoiding the need to re-recalculate full state.
`ValueTask TruncateLogAsync(int beforeVersion)` which would ask the underlying log consistency provider to truncate the log of all events prior to but not including `beforeVersion`, the state should be reloaded from the truncation point before this call returns.
`ValueTask ClearLogAsync()` which would clear all log entries for the grain and recalculate state.
`ValueTask TryClearLogAsync(int version)` which would clear all log entries for the grain and recalculate state if the version of the log matched at the time of call.
I assume to preserve backward compatibility we'd want Log providers to opt-in to support these functions, and therefore we'd need a mechanism for the event sourced grain to determine whether the underlying provider supported them before invoking the methods - any advice on the best approach appreciated.
Some options are:
- all of the methods could return bool to indicate whether the provider supports the operation in its current implementation, state or configuration - this might be useful anyway, if for example a provider can support snapshots if configured a certain way, or only truncate before events are confirmed for example.
- do nothing - the methods could throw a NotSupportedException() or NotImplementedException() and it would be up to developers to pair event sourced grains that used those methods with providers that supported them.
- bool properties on JournaledGrain indicate whether the log provider supports the function e.g. CanSnapshot, CanTruncate, CanClearLog
The actual implementation detail in the log providers is not provided here, as I wanted to approach this from grain developer API perspective first.
Any and all feedback appreciated.
Contributor guide
Research direction
Start with JournaledGrain and the log consistency provider API, then review the proposed SnapshotAsync, TruncateLogAsync, ClearLogAsync, and TryClearLogAsync contracts. Establish provider opt-in behavior and verify snapshotting, truncation, deletion, version checks, state reload, and backward compatibility before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100