dapr / dapr/dotnet-sdk

Sporadic inconsistent actor state is returned through multiple ActorInterfaceMethod and ReminderMethod invocations

Open
#1,616 3 comments 0 reactions 1 assignee Claimed by @WhitWaldo View on GitHub
area/actor/client kind/bug
Dominant language
C#
Stars
1.2k
Forks
378
Avg merge
1d 22h
Merged PRs (30d)
5

Description

I am encountering a sporadic issue with actor's state synchronization and would love any directions or insights

**Overview**
I have an actor that receives a stream of events (via pub-sub)
For each event it will update the actor's state and register `a-reminder`
When `a-reminder` triggers it reads the actor's state and starts processing, then persists the state and registers `b-reminder` to complete the processing
When `b-reminder` triggers it reads the actor's state and completes processing

**Infra**
- Dapr 1.15.4
- `dotnet-sdk` for all components
- `rabbitmq` as pub-sub backend
- `postgresql` as actor-state-store.

**The issue**
Sometimes I get the following flow:
1. `event1` received by actor, `state1` is persisted, and `a-reminder` is registered
2. `a-reminder` triggers, reads `state1`, unregisters itself, persists `state1.1` and registers `b-reminder`
3. `event2` received by actor, `a-reminder` & `b-reminder` are unregistered, `state2` is persisted, a new `a-reminder` is registered
4. `a-reminder` triggers and reads the actor's state instead of the expected `state2` it gets `state1.1`
(even more puzzling to me:
5. `event3` received by actor and sees `state2` in state store )

Worth noting that this is not consistent, sometimes the reminder will get the expected `state2` even though execution order is the same

**What I looked for?**
- tried reproducing with a simpler setup and could not reproduce (yet)
- tried to lookup etag errors from state-store, didn't find anything
- tried to lookup any errors in dapr-sidecar, didn't find anything
- actor logs indicate serial execution of the above flow

## Expected Behavior

State store should either indicate some ETag error on `StateManager.SaveStateAsync()` or consistently the latest save data should return upon `StateManager.GetStateAsync`

## Actual Behavior

Inconsistent state is returned from different reminder invocations

## Steps to Reproduce the Problem

Still trying to create a minimal reproduction

## Release Note

N/A

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.