microsoft / microsoft/durabletask-netherite
Shadow store support
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 239
- Forks
- 37
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 2
Description
Based on repeated user feedback, I am considering adding a feature to support shadow stores.
Configuring a shadow store allows users to mirror the current state of all instances (orchestrations or entities) in an external database. Shadow stores can be useful to support more types of queries (e.g. by using a database that indexes all fields or supports relational queries), or to support a high volume of instance status queries (currently a weakness of Netherite). There may be other uses as well.
The current design sketch is as follows:
- users write a piece of code that knows how to update the external store, and implements an
IShadowStoreinterface, something like
public interface IShadowStore
{
Task OnInstanceUpdated(OrchestrationInstance instanceState);
Task OnInstanceDeleted (string instanceId);
}
- this adapter is registered with the Netherite orchestration service during startup
- there are two new client operations
AddShadowStoreandRemoveShadowStorethat allow any number of shadow stores to be registered with Netherite. - there is a new client operation
GetShadowStoreStatusthat can display the status of a shadow store (e.g. number of pending propagations, error messages).
Once a shadow store is registered, Netherite automatically and persistently calls the IShadowStore methods to ensure the shadow store stays up to date (regardless of crashes, application restarts, etc). It reliably keeps track of any state updates that need to be propagated. Note that it may skip intermediate updates (e.g. if an instance state is updated multiple times it may send only the last update).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the design sketch in this issue, focusing on the IShadowStore interface and the AddShadowStore, RemoveShadowStore, and GetShadowStoreStatus operations. Resolve the registration, persistence, update propagation, crash recovery, and status requirements before implementation. Done means shadow stores reliably reflect instance state across updates, deletions, crashes, and restarts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100