microsoft / microsoft/durabletask-netherite

Shadow store support

Open
#338 2 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
C#
Stars
239
Forks
37
Avg merge
1d 10h
Merged PRs (30d)
2

Description

Based on repeated user feedback, I am considering adding a feature to support shadow stores.

Configuring a shadow store allows users to mirror the current state of all instances (orchestrations or entities) in an external database. Shadow stores can be useful to support more types of queries (e.g. by using a database that indexes all fields or supports relational queries), or to support a high volume of instance status queries (currently a weakness of Netherite). There may be other uses as well.

The current design sketch is as follows:

  • users write a piece of code that knows how to update the external store, and implements an IShadowStore interface, something like
  public interface IShadowStore
  {
      Task OnInstanceUpdated(OrchestrationInstance instanceState);
      Task OnInstanceDeleted (string instanceId);
  }
  • this adapter is registered with the Netherite orchestration service during startup
  • there are two new client operations AddShadowStore and RemoveShadowStore that allow any number of shadow stores to be registered with Netherite.
  • there is a new client operation GetShadowStoreStatus that can display the status of a shadow store (e.g. number of pending propagations, error messages).

Once a shadow store is registered, Netherite automatically and persistently calls the IShadowStore methods to ensure the shadow store stays up to date (regardless of crashes, application restarts, etc). It reliably keeps track of any state updates that need to be propagated. Note that it may skip intermediate updates (e.g. if an instance state is updated multiple times it may send only the last update).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the design sketch in this issue, focusing on the IShadowStore interface and the AddShadowStore, RemoveShadowStore, and GetShadowStoreStatus operations. Resolve the registration, persistence, update propagation, crash recovery, and status requirements before implementation. Done means shadow stores reliably reflect instance state across updates, deletions, crashes, and restarts.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.