dotnet / dotnet/orleans

Make persistent streams rewindable

Open
#756 12 comments 4 reactions 0 assignees View on GitHub
area-streaming enhancement Needs: attention :wave: Needs: design Needs: documentation
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
14h 57m
Merged PRs (30d)
349

Description

Currently the `PersistentStreamProvider` has a preparation for rewinding streams, but it is not implemented in any current StreamProvider. We want to achieve a `PersistentStreamProvider` with full rewinding capabilities that would be able any subscriber to rewind to any event that was saved on the persistent stream (even if it is not currently in the cache).
We offer to make some changes in the current `PersistentStreamProvider` (Or extend it to a new `PersistentStreamProvider`, maybe `RewindableStreamProvider`) model to fit for rewinding. Here is the general description of our plan:

Our plan is to make the `PersistentPullingAgent` hold two types of caches:
- Main Cache - Identical to the cache we currently have in the `PersistentPullingAgent`. The agent will pull batches from the top of the queue using the `IQueueAdapterReceiver` and will put them in this cache.
- Rewinding Caches - These caches will be created when a subscriber wants to subscribe to a certain stream with a `StreamSequenceToken` that is not found in the Main Cache (because it's too old). When this happen, a Rewinding Cache is created which will start from the desired `SequenceToken` and the subscriber will receive its messages from this cache until it "Catches up" to the Main Cache. Once we have a Rewinding Cache, when a subscriber will try to subscribe with a certain `SequenceToken`, then both the Main Cache and the Rewinding Cache will be checked for the wanted `SequenceToken`. If it is not found (too old), then another Rewinding Cache will be created.

Currently we think that both caches can just be based on `SimpleQueueCache`.

The `PersistentPullingAgent` will hold all theses caches. The Rewinding Caches will be sorted from new to old (Caches with newer messages will be before caches with older messages). Then the Agent will have to fill all the caches accordingly and distribute the data between the subscribers.

In order to prevent having too many Rewinding Caches, another thing the `PersistentPullingAgent` will do is to try to move the cursors in older Rewinding Caches to newer ones. Before pulling batches to the caches the Agent will check if there are any overlapping caches in the Rewinding Caches and will transfer the subscribers' cursors to the newer caches if possible (if the cursors are currently in the overlapping part). Then Rewinding Caches without any cursors will be removed. This process will continue until all subscribers have "Caught up" with the Main Cache.

Here are the changes we think needs to be made in order to implement our idea:
- `IQueueAdapterReceiver`
- Adding a `SequenceToken` to `GetQueueMessageAsync`
- `SimpleQueueCache`
- Make the First and Last tokens public
- Add HasCursors method to determine if there are any cursors referencing the cache (which means also keeping reference to the cursors that are referencing to the cache).
- `SimpleQueueCacheCursor`
- Write a `ChangeCache` function
- `PersistentPullingAgent`
- Contain one Main Cache
- ​Contain a collection of caches (the Rewinding Caches)
- Write a `MoveCursorsToNewerCache` method.
- Write `RemoveUnreferencedCaches` method.
- When subscribing to a stream make sure a cursor is created to a relevant cache.
- Updating the `AsyncTimeCallback` method to perform the following actions:
- `GetQueueMessageAsync` calls for the other caches according to their offset
- Call `MoveCursors` and `RemoveUnreferencedCaches` method before starting to update caches.

So this is basically our idea, would love to get some feedbacks about it to know if we are in the right direction, and also to get a discussion going on :smiley:

Contributor guide

Open the contributing guide

Research direction

Start by reading the current PersistentPullingAgent, IQueueAdapterReceiver, SimpleQueueCache, and SimpleQueueCacheCursor implementations to understand cache and cursor behavior. Compare them with the proposed main and rewinding cache model, including cursor movement and cache removal. Done means subscribers can rewind to any event saved on the persistent stream and eventually catch up with the main cache.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.