Data Replication, a.k.a single writer / multi reader pattern

Moving the discussion about this subject from issue #433 .

To summarize so far:

There's a rather basic need for multiple readers / single writer pattern.

This is a common pattern when you have a resource that has high read access. You want the updates to go to a single statefull grain, but to scale reads you want those to be done via `StatelessWorkers` (that also function as a local-silo cache).

These readers needs to subscribe to revoke events from the writer. Doing that via observers is a problem (since you need to persist them etc.).

A possible solution would be to use streams: each `StatelessWorker` will subscribe on activation and will unsubscribe on deactivation. That way stream messages will only go to the existing activations and will not activate any StatelessWorker that was deactivated.

Or, as @veikkoeeva rephrased it better:

You have a `MasterCache` that is an origin, a source of events to some well known stream. Let's say it's a cache stream with Id 0. Then you would like to instantiate local, per-silo caches, `PerSiloCache` grains, that have the exact same content as the master cache grain.

To query these per-silo caches grain you call them with a well-known ID, say 0 (`PerSiloCacheFactory.GetGrain(0)`). Upon `OnActivateAsync` this `PerSiloCache` retrieves data from `MasterCache` and subscribes to the aforementioned well known cache stream with stream ID 0. Upon `DeactivateAsync` you would unsubscribe from the well known cache stream.

This way `MasterCache` could feed changes to `PerSiloCache` grains and you wouldn't need to explicitly manage the caching and/or guess how many you would need.

And @gabikliot wrote:

I see 2 ways to solve your general problem, both don't involve `StatelessWorkers`:
1) Do it the way you are doing now, with a single write grain and a pool of read grains and managed the pool explicitly, via ids. Like you decide on 10 ids for 10 grains for that pool. No need for them to register anywhere, just use known ids 1...10. The readers just picks a random id out of the pol. The write will send invalidate to all of them. To get dynamic scaling, you basically rely on Orleans for short deactivation. Configure the readers to have a short 1 min deactivation, and write have a longer deactivation. So in the high read rate case all 10 grains will be in memory. But in the low period they will quickly be deactivated.
In general, in your case, since you are managing consistency of your data yourself, with explicit invalidate command, this whole problem is relatively easy.

2) We are looking internally on some kind of similar single write multiple reader case. The difference from your case is that we want to manage the consistency automatically, not based on application telling us when to invalidate. This is still in quite an early stage.