Azure / Azure/azure-functions-host

Target-based scaling instance churn increased ~30–100x since 2026-08-18 with no corresponding code/config change (Flex Consumption plan, Node.js)

Open
#11,943 2 comments 0 reactions 0 assignees View on GitHub
Needs: Triage (Functions)
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 10h
Merged PRs (30d)
36

Description

#### Investigative information

Timestamp: 2026-08-18T15:30:51Z (onset), ongoing through 2026-08-25 (further escalation); representative example below at 2026-08-24T12:00:00Z
Function App version: 4.1052.300.26370 (FunctionsExtensionVersion=~4)
Function App name: zoho-functions
Function name(s): Timer A (15-min schedule), Timer B (hourly), Timer C (4h schedule with internal catch-up/due-check logic) — see Invocation IDs below
Invocation ID (all for 2026-08-24T12:00:00Z, same scheduled minute, three separate instances):
Timer A: f64c0039-93cf-4c6d-9747-6c6b8fc1b8b9
Timer B: ec58b4f4-c238-4594-b8fc-65dafb572c0e
Timer C: db5259cc-96bf-4bf2-b8d6-8065d0fb654c
Region: Germany West Central
Hosting plan: Flex Consumption, Instance memory 512MB, Always-ready instances = 0 for all triggers, on-demand maximum instance count 100.

#### Repro steps

No deterministic repro steps — this is a spontaneous, sustained change in production platform behavior, not triggered by a specific code path. Pattern: multiple timer-triggered functions scheduled for the same minute (e.g. 12:00 UTC) each get dispatched to their own separate instance (distinct HostInstanceId, not sharing a process), and each instance is drained and its worker process terminated shortly after completing its single invocation.

#### Expected behavior

Based on our own baseline (2026-08-01 to 08-16): occasional instance recycling (DrainMode mode enabled, category Microsoft.Azure.WebJobs.Host.DrainModeManager) at 1–4 occurrences/day, with drain cycles rarely resulting in a full Language Worker Process exited (worker process typically reused/reset internally instead).

#### Actual behavior

Since 2026-08-18, DrainMode mode enabled frequency increased ~30–100x and has stayed elevated:

- 2026-08-01 to 08-16 (baseline): 1–4/day
- 2026-08-18: 92/day
- 2026-08-19 to 08-24: 238–320/day
- 2026-08-25: 186/day (partial day)

Additionally, starting 2026-08-25, the proportion of drain cycles ending in a full Language Worker Process exited jumped sharply:

- 2026-08-24: 320 DrainMode events, 24 worker exits → 7.5% exit-to-drain ratio
- 2026-08-25 (partial): 186 DrainMode events, 153 worker exits → 82.3% exit-to-drain ratio

We confirmed via a join of all 153 exit events on 2026-08-25 against DrainMode mode enabled (same cloud_RoleInstance, within 5 minutes prior) that 100% were preceded by an orderly drain sequence — no unresponsive-process kills, just far more of them completing all the way to process termination. The 2026-08-25 escalation began between 21:00–23:00 UTC on 08-24 (3 → 11 exits/hour) and reached full elevated levels by 00:00 UTC on 08-25 (18/hour).

Example instance lifecycle for the three invocations listed above (2026-08-24, 12:00 UTC): Timer A runs 18.3s, completes successfully, then ~19s later its instance receives DrainMode mode enabled, cleanly stops all listeners (~100ms), Job host stopped. Timer C completes a trivial 446ms no-op, sits idle 38s, then is drained the same way, ending in Language Worker Process exited. Pid=64. Timer B completes successfully (9.5s), drained ~30s later via the same clean sequence. No exception, no timeout, no unresponsiveness in any case.

We ruled out on our side: our own deployments/merges (none in the relevant windows, or on unrelated feature branches not deployed to production), configuration/Activity Log changes, Azure Service Health incidents, host runtime version changes (single consistent version across 2093 host starts, 08-17 to 08-24), and memory pressure (Max Memory working set constantly ~1GB against configured 512MB for 30+ days with no change around either onset date — pre-existing, unrelated to the timing). A manual restart of the Function App (2026-08-19) did not reduce the churn frequency afterward.

#### Known workarounds

None found. Manual Function App restart did not help.

#### Related information

Programming language: Node.js 22, @azure/functions v4 programming model
Bindings used: Timer trigger (NCRONTAB schedules), also affects HTTP-triggered functions on the same app (elevated error rate on one HTTP function from Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content, consistent with connections aborting during the same instance churn)
Question for the maintainers: is a ~30–100x increase in target-based-scaling instance churn, and the jump from ~7.5% to ~82% in drain cycles ending in full process termination, expected under some condition on Flex Consumption with Always-ready instances = 0, or does this indicate a regression in the scaling controller?

Contributor guide

Open the contributing guide

Research direction

No source file, test, or entry point is named. Start with the supplied DrainMode, worker-exit, and instance-lifecycle telemetry for 2026-08-18–25, then compare it with target-based scaling behavior for Flex Consumption and the reported configuration. Done means identifying whether this is expected behavior or a scaling-controller regression, with evidence for the conclusion.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, node.js
Domain
backend, cloud
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.