Azure / Azure/azure-functions-host

COMException and RpcException in Function App

Open
#4,192 16 comments 1 reaction 0 assignees View on GitHub
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 10h
Merged PRs (30d)
36

Description

We have had a couple of incidents recently where our function apps start erroring badly. We see a high percentage of failures (many 1000's) across all functions in the app caused by **System.Runtime.InteropServices.COMException** and **Microsoft.Azure.WebJobs.Script.Rpc.RpcException**.
The only way we have found to fix the issue is to restart the function app after which point the errors completely stop.
Interestingly, we have seen that this incident seems to affect all function apps running on a given app service plan.
The errors seem to be coming from the function runtime. It would be useful to know if this is caused by instability in the func runtime or if it is something we are doing wrong.

Our function apps are written in Node, run the latest V2 runtime and run on app service plans. Most of our functions are http triggers.

#### Investigative information
The last incident started on 2019-03-10 2100 UTC and lasted around 14 hours (at which point we restarted the app and the errors stopped)

Com Exception Example
- Timestamp: 2019-03-11T09:30:59.8876066Z
- Function App version: 2.0
- Invocation ID: 515cf6a6-79c9-46dc-bb62-db6fb1d02be1
- Region: SEA

Rpc Exception Example
- Timestamp: 2019-03-11T07:50:35.8048046Z
- Function App version: 2.0
- Invocation ID: 148bd6f4-2fda-4658-a446-48eca495451e
- Region: SEA

Contributor guide

Open the contributing guide

Research direction

No source file, test, or entry point is named. Start by reviewing the reported COMException and RpcException events alongside the Function App and App Service plan details, then determine whether the host runtime or application behavior is responsible. Done means identifying a reproducible cause and documenting whether restarting the app is an expected mitigation.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, node.js
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.