PartitionGrains and Rebalancing
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 15h 1m
- Merged PRs (30d)
- 345
Description
Hi, we a have a common scenario in our Orleans setup which involes the use of `{Entity}PartitionGrain`s, those grains are responsible for 100,000s of entities and they forward events over implicit streams to the singular `{Entity}Grain`
Generally the number of partitions is low, e.g. 8, 16, 32
Those `EntityPartitionGrain`s can take many forms, e.g.:
* Each partition has a `Trill` query instance running within it and is responsbile for doing some streaming analytics on its partition of entities before forwarding to `EntityGrain`s.
* Each partition has a `RocksDB` instance which efficiently updates and enriches the entity's data before forwarding to `EntityGrain`s.
* In a multi-tenancy application each `TenantGrain` subscribes to its own `MQTT` topic wildcard e.g. `tenants/{TenantId|GrainId}/*` and forwards payloads to `EntityGrain`s.
This is an example of what the setup looks like:

This system works very well and performantly until we consider scaling.
Ideally, after a new silo joins the cluster, the placement of grains should change to:

There are 2 things at play here:
1. The partition grains themselves are balanced evenly across active silos, since they are generally "heavy" grains.
2. All active entity grains that belong to a certain partition are migrated across with their partition grain
- This can also mean those grains are simply deactivated on the migration of their partition.
- Or they are lazily migrated the next time the partition grain forwards a request to them.
## Considerations
* `EntityGrain`s receive events and calls from other sources, for example they may react to domain events generated by other microservices that update their state, hence why moving those grains into the partition itself is not an option.
## Questions
1. Does the use `PartitonGrain`s align with Orleans design or is it an uncommon use case?
2. Does Orleans support a `TypeActivationCount` placement strategy? Or is it currently feasible to implement as a custom placement strategy? Specifically, is there a way to track the number of activations per silo for a specific grain type?
4. Similarly, in your future plans for rebalancing of grains, (if you have given this some thought), do you envision it supporting balancing based on grain type activation count?
5. Any ideas on how the (partition => grain) association may work when migrating/rebalancing of the partition grain?
Contributor guide
Assessment
This issue has not been assessed yet.