dotnet / dotnet/orleans

PubSub Streams SMS scalability issues

Open
#6,919 4 comments 5 reactions 0 assignees View on GitHub
area-performance
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
14h 42m
Merged PRs (30d)
354

Description

We need some tips in order to make PubSub (SMS) more robust/performant as after some time (3-7 days) it seems we are having issues with the cluster due to this; and we would need to restart and clear state in order to recover.

Generally the issue with scaling is when we have a stream to target multiple grains ("all") such as following:

- `UserGrain` (grain per user)
1. Subscribes to Stream `"invalidate-all"` - *topic which invalidates cache for all users; this kind of stream which will give issues*
2. Subscribes to Stream `"invalidate-user-xx"` *topic which invalidates cache for THIS specific user; this won't give any issues*
- `XyzGrain` (Publisher grain) publishes on `invalidate-all`

The reason why the `invalidate-all` will give issues is because the state of the pubsub state will keep increasing per each user and read/write will be heavy/slow and eventually end up timing out or reaching the size limit (depending what store you use).

Also, is it good to `unsubscribe` for a stream during **grain deactivation**? If im not wrong in some git issue I had read that its not good as it might deadlock (not 100% sure tho); As unsubscribing would definitely help size to remain smaller, we recently had disabled unsubscribing as we were experiencing several timeouts on deactivations during stream cleanup.

Maybe also using different serializer which is more compact/faster would probably help, we had tried that in the past alto with pubsub I believe we were having issues with deserializing; but I can give it a go again (maybe mention a serializer which works well if possible)

Any suggestions how to handle such scenarios with stream? The scenario is a sample, we do have others which is similar as a pattern

We are running on:
- Orleans: 3.4.0
- .net core: 3.1
- state persistance: redis - json

Contributor guide

Open the contributing guide

Research direction

Start with Orleans PubSub Streams and SMS behavior for subscriptions targeting all grains, then inspect how Redis JSON persistence stores and updates subscription state. Reproduce the invalidate-all pattern with UserGrain subscriptions and observe state growth, timeouts, and deactivation cleanup. Done would require a confirmed cause and documented guidance or an agreed implementation scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, redis
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.