Azure / Azure/azure-functions-host

Timer-Triggered Functions Suddenly Stopped Triggering

Open
#8,792 3 comments 0 reactions 1 assignee Assigned to @ramya894 View on GitHub
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 10h
Merged PRs (30d)
36

Description

#### Investigative information

- Timestamp: 2022-09-27T13:55:45.1227038Z
- Function App version: 4.10.4.4
- Function App name: sm-3rdparty-import-functions-test
- Function name(s) (as appropriate):
- Zaptec-StartImportPartition0 ... Zaptec-StartImportPartition59 (again, 60 of those, too)
- Easee-StartImportPartition0 ... Easee-StartImportPartition59 (there's 60 of those)
- Invocation ID:
- 5d94aa5a-c8b8-44cd-bb9e-354456085b27 with timestamp: 2022-09-27 13:53:45.196
`Zaptec-AllMetersOfSingleSmartMeUserImportFunction` function which is executed via one of the timer triggered functions.
- bbe48de37d0f42f091daf964e4e3d0cb with completed timestamp 2022-09-27T13:55:45.1227038Z
`ExecutionID` (as found in the `*Instances` table of the storage the functions uses) of last running function `Zaptec-PartitionImportOrchestrationFunction` started via `Zaptec-StartImportPartition59`
- Region: Switzerland-North

#### Repro steps

This has happened out of of the blue after running the app for about 31 hours since last deployment.
Before, the app ran without issue for months (deployments were dependency updates only).

Therefore, I'm not sure what the steps to reproduce are. I'll give an overview of the application instead:

There's a total of 120 timer triggered functions:

##### Zaptec
The `Zaptec-StartImportPartition*` functions are intended to be run every 15 minutes each. There's 60 functions which have a 15 seconds offset from each other. That is, every 15 seconds one of the `Zaptec*` functions should be run, starting with `Zaptec-StartImportPartition0`, then `Zaptec-StartImportPartition1` after 15 seconds and so on.

All these functions do is using `IDurableOrchestrationClient` to `StartNewAsync` another function called `Zaptec-PartitionImportOrchestrationFunction` passing in the partition number (The `*` in the name of the timer triggered ``Zaptec-StartImportPartition*` function). This function then does the actual work.

##### Easee
The `Easee-StartImportPartition*` functions work similarly to the Zaptec once with the difference that there are 60 functions run once every hour with a 1 minute offset from each other.

#### Expected behavior

The timer triggered functions should continue to be executed periodically.

#### Actual behavior

All timer triggered functions of the function app have stopped executing after 2022-09-27T13:55:45.1227038Z
Function execution count:
![image](https://user-images.githubusercontent.com/2090172/192733548-f5454307-2c3b-48fb-bf10-33ad6459d96a.png)

#### Known workarounds

Possibly restarting the function app.
(I didn't to it because I thought someone might want to investigate first. This happened on our test system so we do have time to investigate further).

#### Related information

- Programming Language: C#
- Package References:
- Microsoft.NET.Sdk.Functions 4.1.3
- Microsoft.Azure.WebJobs.Extensions.DurableTask 2.8.1
- Microsoft.Azure.WebJobs.Extensions.Storage 5.0.1
- Microsoft.ApplicationInsights.WorkerService 2.21.0
- ... and more

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.