What does it mean that one silo does not have another silo in their MembershipOracle?

Open

#7,633 opened on Mar 16, 2022 1 comment 0 reactions 1 assignee Claimed by @ReubenBond View on GitHub

question

Dominant language: C#
Stars: 10.9k
Forks: 2.1k
Avg merge: 14h 42m
Merged PRs (30d): 354

Description

I have a heterogeneous cluster and whenever I scale down everything and start it, it basically start working as expected, but then I try to scale up/down while cluster is running and I keep getting message logs from the "unhealthy" silo like so:

```
[21:33:47 WRN] Did not get response for probe #23404 to silo S10.244.7.127:11111:385044955 after 00:00:03.0244810. Current number of consecutive failed probes is 23323

Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.244.7.127:11111:385044955. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.244.7.127:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
at Orleans.Runtime.MembershipService.SiloHealthMonitor.ProbeDirectly(CancellationToken cancellation)

[21:33:47 DBG] ReadTableVersion is executed.

[21:33:47 DBG] -TryToSuspectOrKill: Read Membership table 44 silos, 6 are Active, 38 are Dead, Version=<10, 5020>. All silos: [SiloAddress=S10.244.4.48:11111:385037826 SiloName=inputprocessor-5f5b89bc-jprb7 Status=Active, SiloAddress=S10.244.5.139:11111:385046333 SiloName=purchaseprocessor-c55f584c6-c2xcd Status=Active, SiloAddress=S10.244.8.111:11111:385046324 SiloName=purchaseprocessor-c55f584c6-n4lv5 Status=Active, SiloAddress=S10.244.2.118:11111:385046329 SiloName=purchaseprocessor-c55f584c6-r6gsv Status=Active, SiloAddress=S10.244.7.131:11111:385046330 SiloName=purchaseprocessor-c55f584c6-vknjh Status=Active, SiloAddress=S10.244.3.90:11111:385046325 SiloName=purchaseprocessor-c55f584c6-x5bff Status=Active, SiloAddress=S10.244.5.134:11111:385044951 SiloName=purchaseprocessor-559597854b-2r6zn Status=Dead, SiloAddress=S10.244.2.117:11111:385046033 SiloName=purchaseprocessor-9887bc887-spwqh Status=Dead, SiloAddress=S10.244.8.110:11111:385045383 SiloName=purchaseprocessor-86677b855d-mb848 Status=Dead, SiloAddress=S10.244.5.137:11111:385046030 SiloName=purchaseprocessor-9887bc887-mbq6l Status=Dead, SiloAddress=S10.244.4.46:11111:384542264 SiloName=inputprocessor-5b8c7dddfb-pc2dn Status=Dead, SiloAddress=S10.244.2.116:11111:385045388 SiloName=purchaseprocessor-86677b855d-wfv6q Status=Dead, SiloAddress=S10.244.7.115:11111:384542710 SiloName=inputprocessor-5b8c7dddfb-kkgwl Status=Dead, SiloAddress=S10.244.5.124:11111:384543618 SiloName=purchaseprocessor-6bf6b7c7cc-79lx8 Status=Dead, SiloAddress=S10.244.7.117:11111:384543621 SiloName=inputprocessor-5b8c7dddfb-jjpmv Status=Dead, SiloAddress=S10.244.2.113:11111:385036392 SiloName=purchaseprocessor-6bf6b7c7cc-7hc57 Status=Dead, SiloAddress=S10.244.8.108:11111:385036387 SiloName=purchaseprocessor-6bf6b7c7cc-sx8nm Status=Dead, SiloAddress=S10.244.7.127:11111:385044955 SiloName=purchaseprocessor-559597854b-ztzbm Status=Dead, SiloAddress=S10.244.5.136:11111:385045392 SiloName=purchaseprocessor-86677b855d-txfzc Status=Dead, SiloAddress=S10.244.7.128:11111:385045390 SiloName=purchaseprocessor-86677b855d-9bjs6 Status=Dead, SiloAddress=S10.244.5.122:11111:384542692 SiloName=purchaseprocessor-6bf6b7c7cc-ndbzv Status=Dead, SiloAddress=S10.244.5.123:11111:384542692 SiloName=purchaseprocessor-6bf6b7c7cc-djpkw Status=Dead, SiloAddress=S10.244.7.129:11111:385046028 SiloName=purchaseprocessor-9887bc887-slsfh Status=Dead, SiloAddress=S10.244.8.103:11111:384542262 SiloName=purchaseprocessor-6bf6b7c7cc-t9gr2 Status=Dead, SiloAddress=S10.244.2.108:11111:384542689 SiloName=purchaseprocessor-6bf6b7c7cc-mcvtr Status=Dead, SiloAddress=S10.244.7.114:11111:384542689 SiloName=purchaseprocessor-6bf6b7c7cc-dfwwv Status=Dead, SiloAddress=S10.244.8.109:11111:385036939 SiloName=purchaseprocessor-579c794cc9-szxsc Status=Dead, SiloAddress=S10.244.2.114:11111:385036492 SiloName=purchaseprocessor-579c794cc9-k2mqd Status=Dead, SiloAddress=S10.244.3.88:11111:385036483 SiloName=purchaseprocessor-579c794cc9-wbnvh Status=Dead, SiloAddress=S10.244.7.125:11111:385036488 SiloName=purchaseprocessor-579c794cc9-2wkj6 Status=Dead, SiloAddress=S10.244.5.135:11111:385044957 SiloName=purchaseprocessor-559597854b-ft4mr Status=Dead, SiloAddress=S10.244.7.126:11111:385044948 SiloName=purchaseprocessor-559597854b-lkn4f Status=Dead, SiloAddress=S10.244.5.133:11111:385036486 SiloName=purchaseprocessor-579c794cc9-mbrsz Status=Dead, SiloAddress=S10.244.2.115:11111:385044954 SiloName=purchaseprocessor-559597854b-8n5r8 Status=Dead, SiloAddress=S10.244.5.138:11111:385046037 SiloName=purchaseprocessor-9887bc887-f72r7 Status=Dead, SiloAddress=S10.244.8.104:11111:384542900 SiloName=purchaseprocessor-6bf6b7c7cc-8lfhs Status=Dead, SiloAddress=S10.244.8.105:11111:384543623 SiloName=purchaseprocessor-6bf6b7c7cc-bvj97 Status=Dead, SiloAddress=S10.244.7.116:11111:384543622 SiloName=purchaseprocessor-6bf6b7c7cc-j8jbt Status=Dead, SiloAddress=S10.244.2.109:11111:384543608 SiloName=purchaseprocessor-6bf6b7c7cc-gjzx7 Status=Dead, SiloAddress=S10.244.7.123:11111:385036396 SiloName=purchaseprocessor-6bf6b7c7cc-rhnks Status=Dead, SiloAddress=S10.244.5.131:11111:385036404 SiloName=purchaseprocessor-6bf6b7c7cc-q59pc Status=Dead, SiloAddress=S10.244.7.124:11111:385036413 SiloName=inputprocessor-5b8c7dddfb-7swkq Status=Dead, SiloAddress=S10.244.3.89:11111:385045384 SiloName=purchaseprocessor-86677b855d-956x8 Status=Dead, SiloAddress=S10.244.7.130:11111:385046035 SiloName=purchaseprocessor-9887bc887-ngqwh Status=Dead]

[21:33:47 DBG] -TryToSuspectOrKill S10.244.5.135:11111:385044957: The current status of S10.244.5.135:11111:385044957 in the table is Dead, its entry is SiloAddress=S10.244.5.135:11111:385044957 SiloName=purchaseprocessor-559597854b-ft4mr Status=Dead

[21:33:47 DBG] ProcessTableUpdate (called from TrySuspectOrKill) membership table 44 silos, 6 are Active, 38 are Dead, Version=<10, 5020>. All silos: [SiloAddress=S10.244.4.48:11111:385037826 SiloName=inputprocessor-5f5b89bc-jprb7 Status=Active, SiloAddress=S10.244.5.139:11111:385046333 SiloName=purchaseprocessor-c55f584c6-c2xcd Status=Active, SiloAddress=S10.244.8.111:11111:385046324 SiloName=purchaseprocessor-c55f584c6-n4lv5 Status=Active, SiloAddress=S10.244.2.118:11111:385046329 SiloName=purchaseprocessor-c55f584c6-r6gsv Status=Active, SiloAddress=S10.244.7.131:11111:385046330 SiloName=purchaseprocessor-c55f584c6-vknjh Status=Active, SiloAddress=S10.244.3.90:11111:385046325 SiloName=purchaseprocessor-c55f584c6-x5bff Status=Active, SiloAddress=S10.244.5.134:11111:385044951 SiloName=purchaseprocessor-559597854b-2r6zn Status=Dead, SiloAddress=S10.244.2.117:11111:385046033 SiloName=purchaseprocessor-9887bc887-spwqh Status=Dead, SiloAddress=S10.244.8.110:11111:385045383 SiloName=purchaseprocessor-86677b855d-mb848 Status=Dead, SiloAddress=S10.244.5.137:11111:385046030 SiloName=purchaseprocessor-9887bc887-mbq6l Status=Dead, SiloAddress=S10.244.4.46:11111:384542264 SiloName=inputprocessor-5b8c7dddfb-pc2dn Status=Dead, SiloAddress=S10.244.2.116:11111:385045388 SiloName=purchaseprocessor-86677b855d-wfv6q Status=Dead, SiloAddress=S10.244.7.115:11111:384542710 SiloName=inputprocessor-5b8c7dddfb-kkgwl Status=Dead, SiloAddress=S10.244.5.124:11111:384543618 SiloName=purchaseprocessor-6bf6b7c7cc-79lx8 Status=Dead, SiloAddress=S10.244.7.117:11111:384543621 SiloName=inputprocessor-5b8c7dddfb-jjpmv Status=Dead, SiloAddress=S10.244.2.113:11111:385036392 SiloName=purchaseprocessor-6bf6b7c7cc-7hc57 Status=Dead, SiloAddress=S10.244.8.108:11111:385036387 SiloName=purchaseprocessor-6bf6b7c7cc-sx8nm Status=Dead, SiloAddress=S10.244.7.127:11111:385044955 SiloName=purchaseprocessor-559597854b-ztzbm Status=Dead, SiloAddress=S10.244.5.136:11111:385045392 SiloName=purchaseprocessor-86677b855d-txfzc Status=Dead, SiloAddress=S10.244.7.128:11111:385045390 SiloName=purchaseprocessor-86677b855d-9bjs6 Status=Dead, SiloAddress=S10.244.5.122:11111:384542692 SiloName=purchaseprocessor-6bf6b7c7cc-ndbzv Status=Dead, SiloAddress=S10.244.5.123:11111:384542692 SiloName=purchaseprocessor-6bf6b7c7cc-djpkw Status=Dead, SiloAddress=S10.244.7.129:11111:385046028 SiloName=purchaseprocessor-9887bc887-slsfh Status=Dead, SiloAddress=S10.244.8.103:11111:384542262 SiloName=purchaseprocessor-6bf6b7c7cc-t9gr2 Status=Dead, SiloAddress=S10.244.2.108:11111:384542689 SiloName=purchaseprocessor-6bf6b7c7cc-mcvtr Status=Dead, SiloAddress=S10.244.7.114:11111:384542689 SiloName=purchaseprocessor-6bf6b7c7cc-dfwwv Status=Dead, SiloAddress=S10.244.8.109:11111:385036939 SiloName=purchaseprocessor-579c794cc9-szxsc Status=Dead, SiloAddress=S10.244.2.114:11111:385036492 SiloName=purchaseprocessor-579c794cc9-k2mqd Status=Dead, SiloAddress=S10.244.3.88:11111:385036483 SiloName=purchaseprocessor-579c794cc9-wbnvh Status=Dead, SiloAddress=S10.244.7.125:11111:385036488 SiloName=purchaseprocessor-579c794cc9-2wkj6 Status=Dead, SiloAddress=S10.244.5.135:11111:385044957 SiloName=purchaseprocessor-559597854b-ft4mr Status=Dead, SiloAddress=S10.244.7.126:11111:385044948 SiloName=purchaseprocessor-559597854b-lkn4f Status=Dead, SiloAddress=S10.244.5.133:11111:385036486 SiloName=purchaseprocessor-579c794cc9-mbrsz Status=Dead, SiloAddress=S10.244.2.115:11111:385044954 SiloName=purchaseprocessor-559597854b-8n5r8 Status=Dead, SiloAddress=S10.244.5.138:11111:385046037 SiloName=purchaseprocessor-9887bc887-f72r7 Status=Dead, SiloAddress=S10.244.8.104:11111:384542900 SiloName=purchaseprocessor-6bf6b7c7cc-8lfhs Status=Dead, SiloAddress=S10.244.8.105:11111:384543623 SiloName=purchaseprocessor-6bf6b7c7cc-bvj97 Status=Dead, SiloAddress=S10.244.7.116:11111:384543622 SiloName=purchaseprocessor-6bf6b7c7cc-j8jbt Status=Dead, SiloAddress=S10.244.2.109:11111:384543608 SiloName=purchaseprocessor-6bf6b7c7cc-gjzx7 Status=Dead, SiloAddress=S10.244.7.123:11111:385036396 SiloName=purchaseprocessor-6bf6b7c7cc-rhnks Status=Dead, SiloAddress=S10.244.5.131:11111:385036404 SiloName=purchaseprocessor-6bf6b7c7cc-q59pc Status=Dead, SiloAddress=S10.244.7.124:11111:385036413 SiloName=inputprocessor-5b8c7dddfb-7swkq Status=Dead, SiloAddress=S10.244.3.89:11111:385045384 SiloName=purchaseprocessor-86677b855d-956x8 Status=Dead, SiloAddress=S10.244.7.130:11111:385046035 SiloName=purchaseprocessor-9887bc887-ngqwh Status=Dead]

[21:33:48 DBG] UpdateRuntimeStatistics from S10.244.3.90:11111:385046325

[21:33:48 DBG] -The given siloAddress S10.244.3.90:11111:385046325 is not registered in this MembershipOracle.

[21:33:48 DBG] -TryToSuspectOrKill: Read Membership table 44 silos, 6 are Active, 38 are Dead, Version=<10, 5020>. All silos: [SiloAddress=S10.244.4.48:11111:385037826 SiloName=inputprocessor-5f5b89bc-jprb7 Status=Active, SiloAddress=S10.244.5.139:11111:385046333 SiloName=purchaseprocessor-c55f584c6-c2xcd Status=Active, SiloAddress=S10.244.8.111:11111:385046324 SiloName=purchaseprocessor-c55f584c6-n4lv5 Status=Active, SiloAddress=S10.244.2.118:11111:385046329 SiloName=purchaseprocessor-c55f584c6-r6gsv Status=Active, SiloAddress=S10.244.7.131:11111:385046330 SiloName=purchaseprocessor-c55f584c6-vknjh Status=Active, SiloAddress=S10.244.3.90:11111:385046325 SiloName=purchaseprocessor-c55f584c6-x5bff Status=Active, SiloAddress=S10.244.5.134:11111:385044951 SiloName=purchaseprocessor-559597854b-2r6zn Status=Dead, SiloAddress=S10.244.2.117:11111:385046033 SiloName=purchaseprocessor-9887bc887-spwqh Status=Dead, SiloAddress=S10.244.8.110:11111:385045383 SiloName=purchaseprocessor-86677b855d-mb848 Status=Dead, SiloAddress=S10.244.5.137:11111:385046030 SiloName=purchaseprocessor-9887bc887-mbq6l Status=Dead, SiloAddress=S10.244.4.46:11111:384542264 SiloName=inputprocessor-5b8c7dddfb-pc2dn Status=Dead, SiloAddress=S10.244.2.116:11111:385045388 SiloName=purchaseprocessor-86677b855d-wfv6q Status=Dead, SiloAddress=S10.244.7.115:11111:384542710 SiloName=inputprocessor-5b8c7dddfb-kkgwl Status=Dead, SiloAddress=S10.244.5.124:11111:384543618 SiloName=purchaseprocessor-6bf6b7c7cc-79lx8 Status=Dead, SiloAddress=S10.244.7.117:11111:384543621 SiloName=inputprocessor-5b8c7dddfb-jjpmv Status=Dead, SiloAddress=S10.244.2.113:11111:385036392 SiloName=purchaseprocessor-6bf6b7c7cc-7hc57 Status=Dead, SiloAddress=S10.244.8.108:11111:385036387 SiloName=purchaseprocessor-6bf6b7c7cc-sx8nm Status=Dead, SiloAddress=S10.244.7.127:11111:385044955 SiloName=purchaseprocessor-559597854b-ztzbm Status=Dead, SiloAddress=S10.244.5.136:11111:385045392 SiloName=purchaseprocessor-86677b855d-txfzc Status=Dead, SiloAddress=S10.244.7.128:11111:385045390 SiloName=purchaseprocessor-86677b855d-9bjs6 Status=Dead, SiloAddress=S10.244.5.122:11111:384542692 SiloName=purchaseprocessor-6bf6b7c7cc-ndbzv Status=Dead, SiloAddress=S10.244.5.123:11111:384542692 SiloName=purchaseprocessor-6bf6b7c7cc-djpkw Status=Dead, SiloAddress=S10.244.7.129:11111:385046028 SiloName=purchaseprocessor-9887bc887-slsfh Status=Dead, SiloAddress=S10.244.8.103:11111:384542262 SiloName=purchaseprocessor-6bf6b7c7cc-t9gr2 Status=Dead, SiloAddress=S10.244.2.108:11111:384542689 SiloName=purchaseprocessor-6bf6b7c7cc-mcvtr Status=Dead, SiloAddress=S10.244.7.114:11111:384542689 SiloName=purchaseprocessor-6bf6b7c7cc-dfwwv Status=Dead, SiloAddress=S10.244.8.109:11111:385036939 SiloName=purchaseprocessor-579c794cc9-szxsc Status=Dead, SiloAddress=S10.244.2.114:11111:385036492 SiloName=purchaseprocessor-579c794cc9-k2mqd Status=Dead, SiloAddress=S10.244.3.88:11111:385036483 SiloName=purchaseprocessor-579c794cc9-wbnvh Status=Dead, SiloAddress=S10.244.7.125:11111:385036488 SiloName=purchaseprocessor-579c794cc9-2wkj6 Status=Dead, SiloAddress=S10.244.5.135:11111:385044957 SiloName=purchaseprocessor-559597854b-ft4mr Status=Dead, SiloAddress=S10.244.7.126:11111:385044948 SiloName=purchaseprocessor-559597854b-lkn4f Status=Dead, SiloAddress=S10.244.5.133:11111:385036486 SiloName=purchaseprocessor-579c794cc9-mbrsz Status=Dead, SiloAddress=S10.244.2.115:11111:385044954 SiloName=purchaseprocessor-559597854b-8n5r8 Status=Dead, SiloAddress=S10.244.5.138:11111:385046037 SiloName=purchaseprocessor-9887bc887-f72r7 Status=Dead, SiloAddress=S10.244.8.104:11111:384542900 SiloName=purchaseprocessor-6bf6b7c7cc-8lfhs Status=Dead, SiloAddress=S10.244.8.105:11111:384543623 SiloName=purchaseprocessor-6bf6b7c7cc-bvj97 Status=Dead, SiloAddress=S10.244.7.116:11111:384543622 SiloName=purchaseprocessor-6bf6b7c7cc-j8jbt Status=Dead, SiloAddress=S10.244.2.109:11111:384543608 SiloName=purchaseprocessor-6bf6b7c7cc-gjzx7 Status=Dead, SiloAddress=S10.244.7.123:11111:385036396 SiloName=purchaseprocessor-6bf6b7c7cc-rhnks Status=Dead, SiloAddress=S10.244.5.131:11111:385036404 SiloName=purchaseprocessor-6bf6b7c7cc-q59pc Status=Dead, SiloAddress=S10.244.7.124:11111:385036413 SiloName=inputprocessor-5b8c7dddfb-7swkq Status=Dead, SiloAddress=S10.244.3.89:11111:385045384 SiloName=purchaseprocessor-86677b855d-956x8 Status=Dead, SiloAddress=S10.244.7.130:11111:385046035 SiloName=purchaseprocessor-9887bc887-ngqwh Status=Dead]

[21:33:48 DBG] -TryToSuspectOrKill S10.244.5.134:11111:385044951: The current status of S10.244.5.134:11111:385044951 in the table is Dead, its entry is SiloAddress=S10.244.5.134:11111:385044951 SiloName=purchaseprocessor-559597854b-2r6zn Status=Dead

[21:33:48 DBG] ProcessTableUpdate (called from TrySuspectOrKill) membership table 44 silos, 6 are Active, 38 are Dead, Version=<10, 5020>. All silos: [SiloAddress=S10.244.4.48:11111:385037826 SiloName=inputprocessor-5f5b89bc-jprb7 Status=Active, SiloAddress=S10.244.5.139:11111:385046333 SiloName=purchaseprocessor-c55f584c6-c2xcd Status=Active, SiloAddress=S10.244.8.111:11111:385046324 SiloName=purchaseprocessor-c55f584c6-n4lv5 Status=Active, SiloAddress=S10.244.2.118:11111:385046329 SiloName=purchaseprocessor-c55f584c6-r6gsv Status=Active, SiloAddress=S10.244.7.131:11111:385046330 SiloName=purchaseprocessor-c55f584c6-vknjh Status=Active, SiloAddress=S10.244.3.90:11111:385046325 SiloName=purchaseprocessor-c55f584c6-x5bff Status=Active, SiloAddress=S10.244.5.134:11111:385044951 SiloName=purchaseprocessor-559597854b-2r6zn Status=Dead, SiloAddress=S10.244.2.117:11111:385046033 SiloName=purchaseprocessor-9887bc887-spwqh Status=Dead, SiloAddress=S10.244.8.110:11111:385045383 SiloName=purchaseprocessor-86677b855d-mb848 Status=Dead, SiloAddress=S10.244.5.137:11111:385046030 SiloName=purchaseprocessor-9887bc887-mbq6l Status=Dead, SiloAddress=S10.244.4.46:11111:384542264 SiloName=inputprocessor-5b8c7dddfb-pc2dn Status=Dead, SiloAddress=S10.244.2.116:11111:385045388 SiloName=purchaseprocessor-86677b855d-wfv6q Status=Dead, SiloAddress=S10.244.7.115:11111:384542710 SiloName=inputprocessor-5b8c7dddfb-kkgwl Status=Dead, SiloAddress=S10.244.5.124:11111:384543618 SiloName=purchaseprocessor-6bf6b7c7cc-79lx8 Status=Dead, SiloAddress=S10.244.7.117:11111:384543621 SiloName=inputprocessor-5b8c7dddfb-jjpmv Status=Dead, SiloAddress=S10.244.2.113:11111:385036392 SiloName=purchaseprocessor-6bf6b7c7cc-7hc57 Status=Dead, SiloAddress=S10.244.8.108:11111:385036387 SiloName=purchaseprocessor-6bf6b7c7cc-sx8nm Status=Dead, SiloAddress=S10.244.7.127:11111:385044955 SiloName=purchaseprocessor-559597854b-ztzbm Status=Dead, SiloAddress=S10.244.5.136:11111:385045392 SiloName=purchaseprocessor-86677b855d-txfzc Status=Dead, SiloAddress=S10.244.7.128:11111:385045390 SiloName=purchaseprocessor-86677b855d-9bjs6 Status=Dead, SiloAddress=S10.244.5.122:11111:384542692 SiloName=purchaseprocessor-6bf6b7c7cc-ndbzv Status=Dead, SiloAddress=S10.244.5.123:11111:384542692 SiloName=purchaseprocessor-6bf6b7c7cc-djpkw Status=Dead, SiloAddress=S10.244.7.129:11111:385046028 SiloName=purchaseprocessor-9887bc887-slsfh Status=Dead, SiloAddress=S10.244.8.103:11111:384542262 SiloName=purchaseprocessor-6bf6b7c7cc-t9gr2 Status=Dead, SiloAddress=S10.244.2.108:11111:384542689 SiloName=purchaseprocessor-6bf6b7c7cc-mcvtr Status=Dead, SiloAddress=S10.244.7.114:11111:384542689 SiloName=purchaseprocessor-6bf6b7c7cc-dfwwv Status=Dead, SiloAddress=S10.244.8.109:11111:385036939 SiloName=purchaseprocessor-579c794cc9-szxsc Status=Dead, SiloAddress=S10.244.2.114:11111:385036492 SiloName=purchaseprocessor-579c794cc9-k2mqd Status=Dead, SiloAddress=S10.244.3.88:11111:385036483 SiloName=purchaseprocessor-579c794cc9-wbnvh Status=Dead, SiloAddress=S10.244.7.125:11111:385036488 SiloName=purchaseprocessor-579c794cc9-2wkj6 Status=Dead, SiloAddress=S10.244.5.135:11111:385044957 SiloName=purchaseprocessor-559597854b-ft4mr Status=Dead, SiloAddress=S10.244.7.126:11111:385044948 SiloName=purchaseprocessor-559597854b-lkn4f Status=Dead, SiloAddress=S10.244.5.133:11111:385036486 SiloName=purchaseprocessor-579c794cc9-mbrsz Status=Dead, SiloAddress=S10.244.2.115:11111:385044954 SiloName=purchaseprocessor-559597854b-8n5r8 Status=Dead, SiloAddress=S10.244.5.138:11111:385046037 SiloName=purchaseprocessor-9887bc887-f72r7 Status=Dead, SiloAddress=S10.244.8.104:11111:384542900 SiloName=purchaseprocessor-6bf6b7c7cc-8lfhs Status=Dead, SiloAddress=S10.244.8.105:11111:384543623 SiloName=purchaseprocessor-6bf6b7c7cc-bvj97 Status=Dead, SiloAddress=S10.244.7.116:11111:384543622 SiloName=purchaseprocessor-6bf6b7c7cc-j8jbt Status=Dead, SiloAddress=S10.244.2.109:11111:384543608 SiloName=purchaseprocessor-6bf6b7c7cc-gjzx7 Status=Dead, SiloAddress=S10.244.7.123:11111:385036396 SiloName=purchaseprocessor-6bf6b7c7cc-rhnks Status=Dead, SiloAddress=S10.244.5.131:11111:385036404 SiloName=purchaseprocessor-6bf6b7c7cc-q59pc Status=Dead, SiloAddress=S10.244.7.124:11111:385036413 SiloName=inputprocessor-5b8c7dddfb-7swkq Status=Dead, SiloAddress=S10.244.3.89:11111:385045384 SiloName=purchaseprocessor-86677b855d-956x8 Status=Dead, SiloAddress=S10.244.7.130:11111:385046035 SiloName=purchaseprocessor-9887bc887-ngqwh Status=Dead]

[21:33:48 DBG] PublishStatistics.

[21:33:48 DBG] -TryToSuspectOrKill S10.244.7.127:11111:385044955: The current status of S10.244.7.127:11111:385044955 in the table is Dead, its entry is SiloAddress=S10.244.7.127:11111:385044955 SiloName=purchaseprocessor-559597854b-ztzbm Status=Dead

[21:33:48 DBG] UpdateRuntimeStatistics from S10.244.4.48:11111:385037826

[21:33:48 INF] Establishing connection to endpoint S10.244.2.115:11111:385044954
[21:33:48 INF] Establishing connection to endpoint S10.244.5.134:11111:385044951
[21:33:48 INF] Establishing connection to endpoint S10.244.7.127:11111:385044955
[21:33:48 DBG] UpdateRuntimeStatistics from S10.244.2.118:11111:385046329
[21:33:48 DBG] -The given siloAddress S10.244.2.118:11111:385046329 is not registered in this MembershipOracle.
[21:33:48 DBG] UpdateRuntimeStatistics from S10.244.8.111:11111:385046324
[21:33:48 DBG] -The given siloAddress S10.244.8.111:11111:385046324 is not registered in this MembershipOracle.
[21:33:48 DBG] UpdateRuntimeStatistics from S10.244.5.139:11111:385046333
[21:33:48 DBG] -The given siloAddress S10.244.5.139:11111:385046333 is not registered in this MembershipOracle.
[21:33:48 DBG] UpdateRuntimeStatistics from S10.244.7.131:11111:385046330
[21:33:48 DBG] -The given siloAddress S10.244.7.131:11111:385046330 is not registered in this MembershipOracle.
```

This is how my cluster membership table looks like:

```
PK: "10.244.5.139-11111-385046333"
*************************** 1. row ***************************
clusterid: "contabo-development-k8s-settlement-dev"
address: "10.244.5.139"
port: 11111
generation: 385046333
faultzone: 0
hostname: "purchaseprocessor-c55f584c6-c2xcd"
iamalivetime: 5249516650784279278
proxyport: 30000
rolename: "Settlement.PurchaseProcessor.Host"
siloname: "purchaseprocessor-c55f584c6-c2xcd"
starttime: 5249515489767818472
status: 3
suspecttimes: "[]"
updatezone: 0
PK: "10.244.7.131-11111-385046330"
*************************** 2. row ***************************
clusterid: "contabo-development-k8s-settlement-dev"
address: "10.244.7.131"
port: 11111
generation: 385046330
faultzone: 0
hostname: "purchaseprocessor-c55f584c6-vknjh"
iamalivetime: 5249516650757507591
proxyport: 30000
rolename: "Settlement.PurchaseProcessor.Host"
siloname: "purchaseprocessor-c55f584c6-vknjh"
starttime: 5249515489739375840
status: 3
suspecttimes: "[]"
updatezone: 0
PK: "10.244.4.48-11111-385037826"
*************************** 3. row ***************************
clusterid: "contabo-development-k8s-settlement-dev"
address: "10.244.4.48"
port: 11111
generation: 385037826
faultzone: 0
hostname: "inputprocessor-5f5b89bc-jprb7"
iamalivetime: 5249516652822582039
proxyport: 30000
rolename: "Settlement.InputProcessor.Host"
siloname: "inputprocessor-5f5b89bc-jprb7"
starttime: 5249515404696536943
status: 3
suspecttimes: "[]"
updatezone: 0
PK: "10.244.3.90-11111-385046325"
*************************** 4. row ***************************
clusterid: "contabo-development-k8s-settlement-dev"
address: "10.244.3.90"
port: 11111
generation: 385046325
faultzone: 0
hostname: "purchaseprocessor-c55f584c6-x5bff"
iamalivetime: 5249516650723178969
proxyport: 30000
rolename: "Settlement.PurchaseProcessor.Host"
siloname: "purchaseprocessor-c55f584c6-x5bff"
starttime: 5249515489691128873
status: 3
suspecttimes: "[]"
updatezone: 0
PK: "10.244.2.118-11111-385046329"
*************************** 5. row ***************************
clusterid: "contabo-development-k8s-settlement-dev"
address: "10.244.2.118"
port: 11111
generation: 385046329
faultzone: 0
hostname: "purchaseprocessor-c55f584c6-r6gsv"
iamalivetime: 5249516650741814883
proxyport: 30000
rolename: "Settlement.PurchaseProcessor.Host"
siloname: "purchaseprocessor-c55f584c6-r6gsv"
starttime: 5249515489726398466
status: 3
suspecttimes: "[]"
updatezone: 0
PK: "10.244.8.111-11111-385046324"
*************************** 6. row ***************************
clusterid: "contabo-development-k8s-settlement-dev"
address: "10.244.8.111"
port: 11111
generation: 385046324
faultzone: 0
hostname: "purchaseprocessor-c55f584c6-n4lv5"
iamalivetime: 5249516650709648484
proxyport: 30000
rolename: "Settlement.PurchaseProcessor.Host"
siloname: "purchaseprocessor-c55f584c6-n4lv5"
starttime: 5249515489671450161
status: 3
suspecttimes: "[]"
updatezone: 0

6 rows in set (0.001 secs)
```

Logs from healthy silos in the cluster show no abnormal activities besides occasional exceptions about dead silo that they still try to talk to for some reason?

```
[21:40:23 ERR] Directory.RegisterAsync S10.244.5.139:11111:385046333*cli/61ebf69d@735be3eb failed.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.244.5.135:11111:385044957. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.244.5.135:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)

[21:40:23 ERR] OnClientRefreshTimer has thrown an exceptions.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.244.5.135:11111:385044957. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.244.5.135:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
at Orleans.Runtime.ClientObserverRegistrar.OnRefreshClients()
```

so my questions are:

1. Why `[21:33:48 DBG] -The given siloAddress S10.244.2.118:11111:385046329 is not registered in this MembershipOracle.` is logged as far as I was able to read the code MembershipOracle should be some snapshot of MembershipTable that updates every now and then (in my case I guess never, maybe I am doing something wrong?)
2. Why do healthy silos keep trying to reconnect/call directory.RegisterAsync on dead silo (its seen from membership table also that it is dead)?
3. Why after `TrySuspectOrKill 10.244.7.127:11111:385044955` few moments later we are trying to establish again connection to it?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

What does it mean that one silo does not have another silo in their MembershipOracle?

Description

Contributor guide

Assessment

Get new issues in your inbox