Grain activation stuck deactivating and migrating
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 14h 42m
- Merged PRs (30d)
- 354
Description
I'm not sure what triggered it, but we've had a couple instances where we have gotten the following error:
- `Failed to forward message {Message} from {OldAddress} to {ForwardingAddress} after {FailedOperation}. Attempt {ForwardCount}`
- Message = ```IsAlwaysInterleave Request [S10.53.6.100:22222:136317860 guild/7ef1a9ab90ff4279bb333440c1906eca]->[S10.53.7.81:22222:136317861 guildboss/4d1f90e806e142aa8055c950cfa1a590] Orleans.Transactions.Abstractions.ITransactionalResourceExtension.Cancel(System.String, System.Guid, System.DateTime, Orleans.Transactions.TransactionalStatus) #6BD591AC4DC4555B[ForwardCount=2]```
- OldAddress = ```[GrainAddress GrainId guildboss/4d1f90e806e142aa8055c950cfa1a590, ActivationId: @e0dfb664efe547ec9e072f1584fd841b, SiloAddress: S10.53.7.81:22222:136317861]```
- ForwardingAdress = ```S10.53.6.100:22222:136317860```
- FailedOperation = ```Migrating to a new location.. Activation [Activation: S10.53.7.81:22222:136317861/guildboss/4d1f90e806e142aa8055c950cfa1a590@e0dfb664efe547ec9e072f1584fd841b#GrainType=GuildBoss Placement=ResourceOptimizedPlacement State=Deactivating] has been deactivating since 04/28/2026 08:01:03 and is likely stuck```
- ForwardCount = ```2```
Other notes:
- The errors didn't resolve until restarting one of the silos in the cluster.
- The error didn't show until 2 hours after the deactivation time mentioned
- The cluster got a bit overloaded around the deactivation time mentioned (timeouts, etc)
- Beginning shortly after the deactivation time, the same message was still failing but was with ```Dropping expired message {Message} at phase {Phase} Message = IsAlwaysInterleave Request [S10.53.6.100:22222:136317860 guild/7ef1a9ab90ff4279bb333440c1906eca]->[S10.53.7.81:22222:136317861 guildboss/4d1f90e806e142aa8055c950cfa1a590] Orleans.Transactions.Abstractions.ITransactionalResourceExtension.Cancel(System.String, System.Guid, System.DateTime, Orleans.Transactions.TransactionalStatus) #6BD591AC4EF01555```
- A reminder was triggering the messages, so they were happening every minute (the period)
Setup:
- Orleans 9.2.1
- DynamoDB membership
- Default grain directory
- Activation reportioning + rebalancing enabled
- 2 silos that had been running for over 12 hours before the time referenced for deactivation
Contributor guide
Research direction
Start by examining the activation migration and deactivation behavior described in the error logs, with attention to message forwarding, expired messages, and the reminder firing every minute. Reproduce under the reported Orleans 9.2.1 setup with activation repartitioning and rebalancing enabled, then use the stuck activation and silo restart symptoms to identify the failure path. Done means explaining why the activation remains deactivating and correcting it without requiring a silo restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, csharp
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100