Azure / Azure/azure-functions-host
Timer-Triggered Functions Suddenly Stopped Triggering
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
#### Investigative information
- Timestamp: 2022-09-27T13:55:45.1227038Z
- Function App version: 4.10.4.4
- Function App name: sm-3rdparty-import-functions-test
- Function name(s) (as appropriate):
- Zaptec-StartImportPartition0 ... Zaptec-StartImportPartition59 (again, 60 of those, too)
- Easee-StartImportPartition0 ... Easee-StartImportPartition59 (there's 60 of those)
- Invocation ID:
- 5d94aa5a-c8b8-44cd-bb9e-354456085b27 with timestamp: 2022-09-27 13:53:45.196
`Zaptec-AllMetersOfSingleSmartMeUserImportFunction` function which is executed via one of the timer triggered functions.
- bbe48de37d0f42f091daf964e4e3d0cb with completed timestamp 2022-09-27T13:55:45.1227038Z
`ExecutionID` (as found in the `*Instances` table of the storage the functions uses) of last running function `Zaptec-PartitionImportOrchestrationFunction` started via `Zaptec-StartImportPartition59`
- Region: Switzerland-North
#### Repro steps
This has happened out of of the blue after running the app for about 31 hours since last deployment.
Before, the app ran without issue for months (deployments were dependency updates only).
Therefore, I'm not sure what the steps to reproduce are. I'll give an overview of the application instead:
There's a total of 120 timer triggered functions:
##### Zaptec
The `Zaptec-StartImportPartition*` functions are intended to be run every 15 minutes each. There's 60 functions which have a 15 seconds offset from each other. That is, every 15 seconds one of the `Zaptec*` functions should be run, starting with `Zaptec-StartImportPartition0`, then `Zaptec-StartImportPartition1` after 15 seconds and so on.
All these functions do is using `IDurableOrchestrationClient` to `StartNewAsync` another function called `Zaptec-PartitionImportOrchestrationFunction` passing in the partition number (The `*` in the name of the timer triggered ``Zaptec-StartImportPartition*` function). This function then does the actual work.
##### Easee
The `Easee-StartImportPartition*` functions work similarly to the Zaptec once with the difference that there are 60 functions run once every hour with a 1 minute offset from each other.
#### Expected behavior
The timer triggered functions should continue to be executed periodically.
#### Actual behavior
All timer triggered functions of the function app have stopped executing after 2022-09-27T13:55:45.1227038Z
Function execution count:

#### Known workarounds
Possibly restarting the function app.
(I didn't to it because I thought someone might want to investigate first. This happened on our test system so we do have time to investigate further).
#### Related information
- Programming Language: C#
- Package References:
- Microsoft.NET.Sdk.Functions 4.1.3
- Microsoft.Azure.WebJobs.Extensions.DurableTask 2.8.1
- Microsoft.Azure.WebJobs.Extensions.Storage 5.0.1
- Microsoft.ApplicationInsights.WorkerService 2.21.0
- ... and more
Contributor guide
Assessment
This issue has not been assessed yet.