Azure / Azure/azure-functions-powershell-worker

SetupWellKnownPaths wwwroot enumeration fails only on PowerShell 7.6 (.NET 10) worker — Directory.EnumerateFiles regression in .NET 10.0.9, fixed in 10.0.11

Open
#1,147 0 comments 0 reactions 0 assignees View on GitHub
Needs: Triage (Functions)
Dominant language
C#
Stars
215
Forks
61
Avg merge
21h 4m
Merged PRs (30d)
6

Description

## Summary

`FunctionLoader.SetupWellKnownPaths` throws `Could not find file 'C:\home\site\wwwroot'` **only on the PowerShell 7.6 worker**, while 7.2 / 7.4 workers on the same stamps are unaffected. The root cause is a behavioral change in `Directory.EnumerateFiles` between .NET 8 and .NET 10 (specifically introduced in .NET **10.0.9**), interacting with certain Azure Files / content-share filesystem drivers. This issue is the root-cause counterpart to the hardening work in #1146 — #1146 makes the enumeration resilient; this issue documents *why the enumeration started failing in the first place* on 7.6.

## Observed failure

```
Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException : Result: Failure
Exception: Could not find file 'C:\home\site\wwwroot'.
Stack: at System.IO.Enumeration.FileSystemEnumerator`1.FindNextEntry()
at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext()
at System.Linq.Enumerable.TryGetFirstNonIterator[TSource](IEnumerable`1 source, Boolean& found)
at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
at Microsoft.Azure.Functions.PowerShellWorker.FunctionLoader.SetupWellKnownPaths(FunctionLoadRequest request, String managedDependenciesPath)
at Microsoft.Azure.Functions.PowerShellWorker.RequestProcessor.SetupAppRootPathAndModulePath(FunctionLoadRequest functionLoadRequest, String managedDependenciesPath)
at Microsoft.Azure.Functions.PowerShellWorker.RequestProcessor.ProcessFunctionLoadRequest(StreamingMessage request)
```

Note the failure is at `FindNextEntry()` / `MoveNext()` — the directory handle **opens and enumeration starts**, then throws mid-iteration. The resolved path is the correct, fully-qualified `C:\home\site\wwwroot`, so this is not a path-computation bug.

## Relevant code

`src/FunctionLoader.cs`:

```csharp
// Resolve the FunctionApp profile path
var options = new EnumerationOptions { MatchCasing = MatchCasing.CaseInsensitive };
var profiles = Directory.EnumerateFiles(FunctionAppRootPath, "profile.ps1", options);
FunctionAppProfilePath = profiles.FirstOrDefault();
```

`Directory.EnumerateFiles` is lazy, so the directory is walked when `FirstOrDefault()` pulls the first entry — which is exactly where the stack trace lands.

## Evidence pointing at a worker-version / runtime change

Telemetry correlation (production, all stamps/regions) of the `SetupWellKnownPaths` failure against each app's own worker-launch line (`...\workers\powershell\\...`):

- **100% of failing apps run the PowerShell 7.6 worker**, which runs on **.NET 10**.
- **No 7.2 / 7.4 app** (which run on .NET 8) appears in the failing set in any time window examined.
- Affected apps span multiple regions (ln1, am2, ty1, db3, zrh) and multiple stamps, i.e. it is not a single degraded stamp.
- On the **same instance during the same startup**, the Functions host reads `C:\home\site\wwwroot\host.json` and indexes functions successfully seconds *before* the worker fails to enumerate the same directory — so the content share is mounted and readable at that moment.
- The worker's own init (`Worker init request completed`) succeeds; only the later `ProcessFunctionLoadRequest` enumeration fails.

## Root cause

This is a regression in **.NET 10.0.9** and is fixed in **.NET 10.0.11**.

**What changed.** Historically, on Windows, `Directory.EnumerateFiles(dir, "profile.ps1")` passed `null` as the `FileName` filter to `NtQueryDirectoryFile` and did the pattern matching in managed code. As a performance optimization, .NET 10 began passing the search pattern to `NtQueryDirectoryFile` as an **OS-level filter** on the first call per directory:

- Optimization: dotnet/runtime [#122947](https://github.com/dotnet/runtime/pull/122947), backported to `release/10.0` in [#127974](https://github.com/dotnet/runtime/pull/127974) — **shipped in 10.0.9**.

**Why it throws.** When the OS-level filter matches **zero** files, most filesystem drivers report `STATUS_NO_SUCH_FILE` (`0xC000000F`), which the enumerator treats as "no matches" and returns an empty sequence. But some drivers — notably the ones backing certain Azure Files / mounted content shares — report the zero-match condition as **`STATUS_OBJECT_NAME_NOT_FOUND` (`0xC0000034`)** instead. In 10.0.9/10.0.10 the enumerator did not handle that status, so it fell through and threw. `RtlNtStatusToDosError(0xC0000034)` maps to `ERROR_FILE_NOT_FOUND` (2), which surfaces as `FileNotFoundException: "Could not find file ''."` — exactly the exception and message seen here, naming the directory itself.

- Fix: dotnet/runtime [#130196](https://github.com/dotnet/runtime/pull/130196) (adds handling for `STATUS_OBJECT_NAME_NOT_FOUND`), backported to `release/10.0` in [#130327](https://github.com/dotnet/runtime/pull/130327), milestone **10.0.11**. Original .NET report: [dotnet/runtime#130134](https://github.com/dotnet/runtime/issues/130134).

## How this explains the symptoms

- **7.6-only.** Only the 7.6 worker runs on .NET 10; 7.2/7.4 run on .NET 8, which never passes an OS-level filter and is therefore unaffected. The failing population being 100% 7.6 is a direct consequence.
- **Throws mid-iteration at `FindNextEntry()`/`MoveNext()`.** The directory handle opens fine; the throw comes from the first `NtQueryDirectoryFile` filtered query returning `0xC0000034`, which is pulled lazily by `FirstOrDefault()`.
- **`FileNotFoundException` naming the directory, not `DirectoryNotFoundException`.** This corresponds to `ERROR_FILE_NOT_FOUND` (2), which is the mapping for `STATUS_OBJECT_NAME_NOT_FOUND` — i.e. a zero-match filtered enumeration, not a missing/unmounted directory.
- **Host can read the share moments earlier.** Consistent with the above: the mount is visible and readable; the failure is purely how a zero-match filtered enumeration is surfaced by the driver on .NET 10.9/10.10.
- **Intermittent across stamps.** Whether the throw occurs depends on the specific content-share driver returning `0xC0000034` vs `0xC000000F` for the zero-match case, which is why it is not tied to a single degraded stamp and why only apps whose `wwwroot` has no `profile.ps1` (a zero-match) are exposed.

## Path forward

- The .NET runtime available to the PowerShell worker is sourced from the Functions platform, not bundled by this worker. Because the worker is framework-dependent, it runs on whichever .NET 10 patch the platform provides. The fix therefore lands when the platform's .NET 10 runtime is updated to **10.0.11 or later**; we are working with the platform team to pick that up.
- In the meantime, #1146 tracks worker-side hardening of `SetupWellKnownPaths` (existence check + not caching a transient/enumeration error as a terminating failure) so the worker is resilient regardless of the underlying runtime behavior.
- Customer/immediate mitigation: move the app to **PowerShell 7.4** (which runs on .NET 8 and is unaffected) until the platform's .NET 10 runtime is updated. Note 7.6 is still a preview runtime.

## Related

- #1146 — defensive hardening of `SetupWellKnownPaths`.

## Environment

- Worker: PowerShell 7.6 (.NET 10)
- Unaffected: PowerShell 7.4 (.NET 8), 7.2
- Regressed .NET runtime: 10.0.9 and 10.0.10; fixed in 10.0.11

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with src/FunctionLoader.cs and the SetupWellKnownPaths enumeration, then review the linked .NET runtime changes and issue #1146. The issue does not define a worker-side implementation; done is platform confirmation of .NET 10.0.11 or later, or coordination with #1146 for resilience work.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, powershell
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.