dotnet / dotnet/orleans

UpdateIAmAlive retries indefinitely even though it won't be able to succeed

Open
#7,784 17 comments 1 reaction 1 assignee View on GitHub

@ReubenBond is already working on this.

Since Jun 9, 2022.

enhancement
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
13h 56m
Merged PRs (30d)
351

Description

I had this issue with Redis clustering but I believe it may be more generic

**Steps to reproduce**:
1. Spin up a typical cluster with **non**-localhost clustering and let it operate for some time
2. Make an entry for one of the silos disappear from clustering storage without touching the cluster itself. One possibility to do so is to DELETE a row from the table if ADO.NET clustering is used. The other possibility (which happens in my case) is to have Redis clustering with **no** 'Redis persistence' and simply reboot this Redis instance
3. Observe exceptions like this happening every few seconds till the end of times
`Failed to update table entry for this silo, will retry shortly: "Orleans.Clustering.Redis.RedisClusteringException: Could not find a value for the key S10.0.1.101:8952:392386255
at Orleans.Clustering.Redis.RedisMembershipTable.UpdateIAmAlive(MembershipEntry entry)
at Orleans.Runtime.MembershipService.MembershipTableManager.UpdateIAmAlive() in /_/src/Orleans.Runtime/MembershipService/MembershipTableManager.cs:line 201
at Orleans.Runtime.MembershipService.MembershipAgent.UpdateIAmAlive() in /_/src/Orleans.Runtime/MembershipService/MembershipAgent.cs:line 72"`

**Actual result**:
`Orleans.Runtime.MembershipService.MembershipAgent` enters a loop which it can't resolve itself

**Expected result**:
`Orleans.Runtime.MembershipService.MembershipAgent` is able to understand permanent failures (in my example the error message contains `Could not find a value for the key`) and does not retry those (in my example it explicitly states that it will: `will retry shortly`) but instead does something else like trying to reconfigure the whole cluster or consider itself a newly joining silo

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.