microsoft / microsoft/foundry-local

ObjectDisposedException fail-fast in DownloadAndRegisterEpsAsync progress callback (Progress<T> on ThreadPool touches disposed CTS) — WinML 1.2.3

Open
#867 3 comments 0 reactions 1 assignee Claimed by @bmehta001 View on GitHub
Dominant language
C++
Stars
2.6k
Forks
369
Avg merge
2d 17h
Merged PRs (30d)
39

Description

## Summary

An intermittent, **unhandled** `System.ObjectDisposedException: The CancellationTokenSource has been disposed.` is thrown on a **ThreadPool** thread from inside the SDK's own progress-callback marshaling during `FoundryLocalManager.DownloadAndRegisterEpsAsync(...)`. Because it is unhandled on a ThreadPool thread, it **fail-fasts the entire host process** (`0xc0000409`). The calling application has no way to catch it.

## Environment

- Package: `Microsoft.AI.Foundry.Local.WinML` **1.2.3** — faulting module `Microsoft.AI.Foundry.Local.Core.dll`, assembly/file version **1.2.0.0**
- Host: WinUI 3 / Windows App SDK desktop app targeting `net10.0-windows` (the `.NET Runtime` crash event reported `CoreCLR Version: 9.0.17`)
- OS: Windows 11 Pro, build 26200 (x64)
- GPU / EP: NVIDIA GeForce RTX 5090 (CUDA execution provider)

## What the app was doing

One-time initialization. After `FoundryLocalManager.CreateAsync(...)` we register the hardware execution providers with a progress callback, then fetch the catalog:

```csharp
await manager.DownloadAndRegisterEpsAsync(
(epName, pct) => { /* report progress to the UI */ },
cancellationToken);
```

The crash occurs **during** this call. It is **intermittent** — in our logs, EP registration completed successfully **23 times** and fail-fasted **once**. It appears most likely when the EPs are already cached, i.e. the `download_and_register_eps` command returns very quickly.

## Crash details

**Windows Error Reporting — Application Error (Event ID 1000):**

```
Faulting application name: .exe
Faulting module name: Microsoft.AI.Foundry.Local.Core.dll, version: 1.2.0.0
Exception code: 0xc0000409
```

**`.NET Runtime` (Event ID 1026):**

```
Description: The process was terminated due to an unhandled exception.
Exception Info: System.ObjectDisposedException: The CancellationTokenSource has been disposed.
at Microsoft.AI.Foundry.Local.NativeInterop.<>c__DisplayClass13_0.b__9(String, Double)
at System.Progress`1.InvokeHandlers(Object)
at System.Threading.QueueUserWorkItemCallbackDefaultContext`1.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.WindowsThreadPool.DispatchCallback(IntPtr, IntPtr, IntPtr)
```

## Root cause (analysis)

`ExecuteCommandWithCallbackManaged` marshals native progress updates through a `System.Progress`, whose handler invocations are posted to the **ThreadPool** (the `Progress` is constructed on a background thread with no `SynchronizationContext`). The command also uses a per-command `CancellationTokenSource`.

When the `download_and_register_eps` command finishes — quickly, when EPs are already cached — the SDK **disposes that CTS**, but a progress callback (`b__9`) is still queued on the ThreadPool. When that queued callback runs, it touches the now-disposed CTS, throwing `ObjectDisposedException`. Since it runs as a ThreadPool work item with no handler, the runtime terminates the process.

The throw happens **inside the SDK's own wrapper** (`b__9`), *before* control reaches the app-supplied `(epName, pct)` callback, so the calling application cannot observe or handle it (it is not surfaced through `AppDomain.UnhandledException` in a way that can prevent termination).

## Note: appears already addressed on `main`

This managed-interop code path does not appear to exist in the current open-source tree — `ExecuteCommandWithCallbackManaged`, `ExecuteWithTracker`, `ExecuteCommandManaged`, and a `Progress`-based `NativeInterop` all return no hits — and the current EP-download paths use disposal-safe patterns:

- `sdk/cs` routes EP progress through a plain string `ICoreInterop.CallbackFn` (no `Progress`).
- `sdk_v2/cs` checks `ct?.IsCancellationRequested` (safe after dispose) inside the native callback.

So this looks specific to the compiled **1.2.x WinML** assembly and likely resolved by the SDK v2 / unified-package refactor.

## Ask

1. Can you confirm this `Progress` EP-progress / disposed-CTS race is gone in the unified/v2 package?
2. Would you consider a **1.2.x servicing backport** — e.g. guard the progress handler so it cannot touch the CTS after disposal, or defer disposing the per-command CTS until all queued `Progress` callbacks have drained?

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.