Azure / Azure/azure-functions-host

Function metadata load errors when app instance is under resource pressure

Open
#7,238 5 comments 1 reaction 1 assignee Claimed by @pragnagopa View on GitHub
needs-investigation
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

I'm frequently running into the following error when running my function app in my Kubernetes cluster.

```
fail: Function.HelloSequence[3]
Executed 'HelloSequence' (Failed, Id=1c29a263-870f-4621-989a-14d43f0a3380, Duration=0ms)
System.InvalidOperationException: Unable to load metadata for function 'HelloSequence'.
at Microsoft.Azure.WebJobs.Script.WebHost.Diagnostics.FunctionInstanceLogger.StartFunction(FunctionInstanceLogEntry item) in /src/azure-functions-host/src/WebJobs.Script.WebHost/Diagnostics/FunctionInstance
Logger.cs:line 124
at Microsoft.Azure.WebJobs.Script.WebHost.Diagnostics.FunctionInstanceLogger.AddAsync(FunctionInstanceLogEntry item, CancellationToken cancellationToken) in /src/azure-functions-host/src/WebJobs.Script.WebH
ost/Diagnostics/FunctionInstanceLogger.cs:line 88
at Microsoft.Azure.WebJobs.Host.Loggers.CompositeFunctionEventCollector.AddAsync(FunctionInstanceLogEntry item, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azur
e.WebJobs.Host\Loggers\CompositeFunctionEventCollector.cs:line 21
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure
.WebJobs.Host\Executors\FunctionExecutor.cs:line 94
```

I spoke with @balag0 about this and we both suspect it's a race condition. The Kubernetes cluster I'm running in is resource constrained, which likely explains why we'd see this kind of problem there and not in proper Azure Functions environments.

### Investigative information

Please provide the following:

- Docker image: **mcr.microsoft.com/azure-functions/dotnet:3.0**
- Function App version: **3.0.15277**
- Function App name: **N/A**
- Function name(s) (as appropriate): **N/A**
- Invocation ID: **N/A**
- Region: **N/A**

### Repro steps

1. Create a .NET function app and deploy it to a small-ish K8s environment (2 VMs)
2. Add enough load to scale the function app to 5-10 instances (using KEDA)

### Expected behavior

We should never see this error. I verified there are no problems with the Function metadata.

### Actual behavior

See the description above. The issue is inconsistent, which is why I think it's a race condition, probably on app startup.

### Known workarounds

Run on faster VMs?

### Related information

Provide any related information

* Programming language used: **.NET 3.1 Precompiled**
* Links to source: **Probably not relevant**
* Bindings used: **Durable Functions**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.