Spikes of exceptions with "error getting actor state: context deadline exceeded"
- Dominant language
- C#
- Stars
- 1.2k
- Forks
- 378
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 6
Description
Hello, we are using Dapr.Actors.AspNetCore version 1.11.0, Dapr version 1.10.5, and running in the AKS cluster, in which the k8s version is 1.25.5. For storing the state we are using Azure CosmosDB.
Recently, we started receiving alerts for 2 types of exceptions: DaprApiException and ActorInvocationException. All the exceptions have the same message "error getting actor state: context deadline exceeded". The length of the timeframe, in which all the exceptions occur, is always equal to or close to 15 minutes.
Spike only happens to one pod at a time. It means that we can have 3-5 pods with daprd and BE containers, but only one of them will suffer from DaprApiException.
ActorInvocationException can be thrown from lots of different pods during one spike, but services that throw this type of exception are stateless and have no affinities for requests. The exception is thrown in the actor's client object.
Daprd container logs do not contain any errors, except for timer invoking\executing failures.
CosmosDB metrics have shown that DB was 100% available through each 15-minute window. Although, we have observed spikes in max RU consumption from 30-40% up to 100% in the middle of the timeframe. Throttled requests have also been observed at the moment of RU spikes.
The issue has been observed every day through Oct 18-20
Could you please provide more info on what "context deadline exceeded" and if it is somehow related to CosmosDB? If it is indeed related to CosmosDB, then why did it only happen to 1 particular pod and not all of them?
Is "context deadline exceeded" just a timeout or it is something more complex?
Contributor guide
Research direction
Start by correlating the DaprApiException and ActorInvocationException reports with daprd timer failures and the Azure CosmosDB RU and throttling metrics for the 15-minute windows. Review the Dapr.Actors.AspNetCore 1.11.0 and Dapr 1.10.5 behavior around actor state requests. Done means explaining whether CosmosDB throttling causes the timeout and why one pod is affected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp, kubernetes
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100