Azure / Azure/azure-functions-host

"Collection modified" exceptions in WorkerProcess.Exited handler

Open
#7,325 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

Seeing the below exception in production logs. Looks like the issue is in the code [here](https://github.com/Azure/azure-functions-host/blob/4c4fb7da51c593467442fb8570596d575cf94c88/src/WebJobs.Script/Workers/ProcessManagement/WorkerProcess.cs#L138). Should probably make this a ConcurrentQueue or handle concurrency on it some other way.

```
System.InvalidOperationException: Collection was modified after the enumerator was instantiated.
at System.Collections.Generic.Queue`1.Enumerator.MoveNext()
at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
at System.String.Join(String separator, IEnumerable`1 values)
at Microsoft.Azure.WebJobs.Script.Workers.WorkerProcess.OnProcessExited(Object sender, EventArgs e) in /src/azure-functions-host/src/WebJobs.Script/Workers/ProcessManagement/WorkerProcess.cs:line 138
at Microsoft.Azure.WebJobs.Script.Workers.WorkerProcess.b__20_2(Object sender, EventArgs e) in /src/azure-functions-host/src/WebJobs.Script/Workers/ProcessManagement/WorkerProcess.cs:line 58
at System.Diagnostics.Process.OnExited()
at System.Diagnostics.Process.RaiseOnExited()
at System.Diagnostics.Process.CompletionCallback(Object waitHandleContext, Boolean wasSignaled)
at System.Threading._ThreadPoolWaitOrTimerCallback.WaitOrTimerCallback_Context(Object state, Boolean timedOut)
at System.Threading._ThreadPoolWaitOrTimerCallback.WaitOrTimerCallback_Context_f(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(_ThreadPoolWaitOrTimerCallback helper, Boolean timedOut)
```

Contributor guide

Open the contributing guide

Research direction

Start in src/WebJobs.Script/Workers/ProcessManagement/WorkerProcess.cs at OnProcessExited, especially line 138 and the process-exit callback registered near line 58. Inspect how the queue is read and modified during concurrent process-exit handling. Done means the handler no longer throws Collection was modified exceptions under concurrency, with the relevant tests or test suite passing.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.