dotnet / dotnet/orleans

Grain activation stuck deactivating and migrating

Open
#10,044 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
14h 42m
Merged PRs (30d)
354

Description

I'm not sure what triggered it, but we've had a couple instances where we have gotten the following error:

- `Failed to forward message {Message} from {OldAddress} to {ForwardingAddress} after {FailedOperation}. Attempt {ForwardCount}`
- Message = ```IsAlwaysInterleave Request [S10.53.6.100:22222:136317860 guild/7ef1a9ab90ff4279bb333440c1906eca]->[S10.53.7.81:22222:136317861 guildboss/4d1f90e806e142aa8055c950cfa1a590] Orleans.Transactions.Abstractions.ITransactionalResourceExtension.Cancel(System.String, System.Guid, System.DateTime, Orleans.Transactions.TransactionalStatus) #6BD591AC4DC4555B[ForwardCount=2]```
- OldAddress = ```[GrainAddress GrainId guildboss/4d1f90e806e142aa8055c950cfa1a590, ActivationId: @e0dfb664efe547ec9e072f1584fd841b, SiloAddress: S10.53.7.81:22222:136317861]```
- ForwardingAdress = ```S10.53.6.100:22222:136317860```
- FailedOperation = ```Migrating to a new location.. Activation [Activation: S10.53.7.81:22222:136317861/guildboss/4d1f90e806e142aa8055c950cfa1a590@e0dfb664efe547ec9e072f1584fd841b#GrainType=GuildBoss Placement=ResourceOptimizedPlacement State=Deactivating] has been deactivating since 04/28/2026 08:01:03 and is likely stuck```
- ForwardCount = ```2```

Other notes:

- The errors didn't resolve until restarting one of the silos in the cluster.
- The error didn't show until 2 hours after the deactivation time mentioned
- The cluster got a bit overloaded around the deactivation time mentioned (timeouts, etc)
- Beginning shortly after the deactivation time, the same message was still failing but was with ```Dropping expired message {Message} at phase {Phase} Message = IsAlwaysInterleave Request [S10.53.6.100:22222:136317860 guild/7ef1a9ab90ff4279bb333440c1906eca]->[S10.53.7.81:22222:136317861 guildboss/4d1f90e806e142aa8055c950cfa1a590] Orleans.Transactions.Abstractions.ITransactionalResourceExtension.Cancel(System.String, System.Guid, System.DateTime, Orleans.Transactions.TransactionalStatus) #6BD591AC4EF01555```
- A reminder was triggering the messages, so they were happening every minute (the period)

Setup:

- Orleans 9.2.1
- DynamoDB membership
- Default grain directory
- Activation reportioning + rebalancing enabled
- 2 silos that had been running for over 12 hours before the time referenced for deactivation

Contributor guide

Open the contributing guide

Research direction

Start by examining the activation migration and deactivation behavior described in the error logs, with attention to message forwarding, expired messages, and the reminder firing every minute. Reproduce under the reported Orleans 9.2.1 setup with activation repartitioning and rebalancing enabled, then use the stuck activation and silo restart symptoms to identify the failure path. Done means explaining why the activation remains deactivating and correcting it without requiring a silo restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, csharp
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.