Azure / Azure/azure-functions-host
"Collection modified" exceptions in WorkerProcess.Exited handler
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
Seeing the below exception in production logs. Looks like the issue is in the code [here](https://github.com/Azure/azure-functions-host/blob/4c4fb7da51c593467442fb8570596d575cf94c88/src/WebJobs.Script/Workers/ProcessManagement/WorkerProcess.cs#L138). Should probably make this a ConcurrentQueue or handle concurrency on it some other way.
```
System.InvalidOperationException: Collection was modified after the enumerator was instantiated.
at System.Collections.Generic.Queue`1.Enumerator.MoveNext()
at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
at System.String.Join(String separator, IEnumerable`1 values)
at Microsoft.Azure.WebJobs.Script.Workers.WorkerProcess.OnProcessExited(Object sender, EventArgs e) in /src/azure-functions-host/src/WebJobs.Script/Workers/ProcessManagement/WorkerProcess.cs:line 138
at Microsoft.Azure.WebJobs.Script.Workers.WorkerProcess.b__20_2(Object sender, EventArgs e) in /src/azure-functions-host/src/WebJobs.Script/Workers/ProcessManagement/WorkerProcess.cs:line 58
at System.Diagnostics.Process.OnExited()
at System.Diagnostics.Process.RaiseOnExited()
at System.Diagnostics.Process.CompletionCallback(Object waitHandleContext, Boolean wasSignaled)
at System.Threading._ThreadPoolWaitOrTimerCallback.WaitOrTimerCallback_Context(Object state, Boolean timedOut)
at System.Threading._ThreadPoolWaitOrTimerCallback.WaitOrTimerCallback_Context_f(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(_ThreadPoolWaitOrTimerCallback helper, Boolean timedOut)
```
Contributor guide
Research direction
Start in src/WebJobs.Script/Workers/ProcessManagement/WorkerProcess.cs at OnProcessExited, especially line 138 and the process-exit callback registered near line 58. Inspect how the queue is read and modified during concurrent process-exit handling. Done means the handler no longer throws Collection was modified exceptions under concurrency, with the relevant tests or test suite passing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100