dotnet / dotnet/orleans

NotImplementedException in SystemTargetBasedMembershipTable

Open
#9,060 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
14h 42m
Merged PRs (30d)
354

Description

I sometimes hit the following line during local testing on my machine with a development localhost cluster:
https://github.com/dotnet/orleans/blob/8035ed932d0785950410cee348bb2a3d3c41b81d/src/Orleans.Runtime/MembershipService/SystemTargetBasedMembershipTable.cs#L110

I could not yet figure out which situations lead to this behavior, as it only happens sometimes.

I assume, since it is in-memory, that a simple ``return Task.CompletedTask`` would be sufficient here. It should basically be a no-op for a 1-node localhost cluster, right?

Here is a callstack:
``> Orleans.Runtime.dll!Orleans.Runtime.MembershipService.SystemTargetBasedMembershipTable.CleanupDefunctSiloEntries(System.DateTimeOffset beforeDate) Line 110 C#
Orleans.Runtime.dll!Orleans.Runtime.MembershipService.MembershipTableCleanupAgent.CleanupDefunctSilos() Line 71 C#
[Resuming Async Method]
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__6>.ExecutionContextCallback(object s) Line 131 C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 138 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__6>.MoveNext(System.Threading.Thread threadPoolThread) Line 163 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__6>.MoveNext() Line 146 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask) Line 140 C#
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 162 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2850 C#
Orleans.Runtime.dll!Orleans.Runtime.AsyncTimer.NextTick(System.TimeSpan? overrideDelay) Line 88 C#
[Resuming Async Method]
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 138 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__8>.MoveNext(System.Threading.Thread threadPoolThread) Line 163 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask) Line 140 C#
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 162 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2850 C#
System.Private.CoreLib.dll!System.Threading.Tasks.TaskFactory.CompleteOnInvokePromise.Invoke(System.Threading.Tasks.Task completingTask) Line 1000 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2741 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.DelayPromise.CompleteTimedOut() Line 679 C#
System.Private.CoreLib.dll!System.Threading.TimerQueueTimer.Fire(bool isThreadPool) Line 259 C#
System.Private.CoreLib.dll!System.Threading.TimerQueue.FireNextTimers() Line 218 C#
System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 690 C#
System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 1328 C#
[Async Call Stack]
[Async] System.Private.CoreLib.dll!System.Threading.Tasks.Task.Run C#
``

Contributor guide

Open the contributing guide

Research direction

Start at src/Orleans.Runtime/MembershipService/SystemTargetBasedMembershipTable.cs line 110 and trace the call from MembershipTableCleanupAgent.CleanupDefunctSilos using the provided call stack. Reproduce the intermittent failure with a development localhost cluster, determine when cleanup is invoked, and verify that the one-node in-memory membership table completes cleanup without throwing.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.