Sporadic inconsistent actor state is returned through multiple ActorInterfaceMethod and ReminderMethod invocations
- Dominant language
- C#
- Stars
- 1.2k
- Forks
- 378
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 5
Description
I am encountering a sporadic issue with actor's state synchronization and would love any directions or insights
**Overview**
I have an actor that receives a stream of events (via pub-sub)
For each event it will update the actor's state and register `a-reminder`
When `a-reminder` triggers it reads the actor's state and starts processing, then persists the state and registers `b-reminder` to complete the processing
When `b-reminder` triggers it reads the actor's state and completes processing
**Infra**
- Dapr 1.15.4
- `dotnet-sdk` for all components
- `rabbitmq` as pub-sub backend
- `postgresql` as actor-state-store.
**The issue**
Sometimes I get the following flow:
1. `event1` received by actor, `state1` is persisted, and `a-reminder` is registered
2. `a-reminder` triggers, reads `state1`, unregisters itself, persists `state1.1` and registers `b-reminder`
3. `event2` received by actor, `a-reminder` & `b-reminder` are unregistered, `state2` is persisted, a new `a-reminder` is registered
4. `a-reminder` triggers and reads the actor's state instead of the expected `state2` it gets `state1.1`
(even more puzzling to me:
5. `event3` received by actor and sees `state2` in state store )
Worth noting that this is not consistent, sometimes the reminder will get the expected `state2` even though execution order is the same
**What I looked for?**
- tried reproducing with a simpler setup and could not reproduce (yet)
- tried to lookup etag errors from state-store, didn't find anything
- tried to lookup any errors in dapr-sidecar, didn't find anything
- actor logs indicate serial execution of the above flow
## Expected Behavior
State store should either indicate some ETag error on `StateManager.SaveStateAsync()` or consistently the latest save data should return upon `StateManager.GetStateAsync`
## Actual Behavior
Inconsistent state is returned from different reminder invocations
## Steps to Reproduce the Problem
Still trying to create a minimal reproduction
## Release Note
N/A
Contributor guide
Assessment
This issue has not been assessed yet.