Azure / Azure/azure-functions-host

Likely race condition in FileTraceWriter

Open
#1,640 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

A user on Azure Functions doing a high throughput event hubs scenario reported they often see an exception with the following trace:

```
System.ArgumentOutOfRangeException: Index and count must refer to a location within the buffer. Parameter name: chars at
System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32
charCount, Byte[] bytes, Int32 byteIndex, Boolean flush) at
System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder) at
System.IO.StreamWriter.Flush() at
Microsoft.Azure.WebJobs.Host.CompositeTraceWriter.Flush() at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.d__14.MoveNext()

--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task
task) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.d__12.MoveNext()
```
(original report [here](https://social.msdn.microsoft.com/Forums/en-US/baaa5a7b-c662-4909-b80c-21274ea0e1a6/argumentoutofrangeexception-in-eventhub-triggered-function?forum=AzureFunctions))

From the trace it looks likely that the problem is unsynchronized access to a StreamWriter. Considering that FileTraceWriter has the only non-trivial implementation of TraceWriter.Flush, its the most likely source of the issue.

Contributor guide

Open the contributing guide

Research direction

Start by locating FileTraceWriter and its TraceWriter.Flush implementation, then inspect how its StreamWriter is accessed during concurrent function execution. Reproduce the reported flush exception under concurrent writes and verify that the failure no longer occurs.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.