Collectible ALC never collected after Unload() — need guidance on diagnosing root cause

## Environment

- .NET 10.0.0 (`net10.0`), Windows 11
- ASP.NET Core Blazor Server, EF Core 10.0.0 with SqlServer, Microsoft.Data.SqlClient, System.Text.Json, Microsoft.JSInterop
- Diagnosed with `dotnet-dump collect` (full process dump) + `dotnet-dump analyze` following [the unloadability debugging guide](https://learn.microsoft.com/en-us/dotnet/standard/assembly/unloadability#debug-unloading-issues)

## Architecture

We have a modular Blazor Server ERP application. Each module (chat, billing, inventory, HR, etc.) is a separate assembly loaded at runtime into a **collectible** `AssemblyLoadContext`.

### How module loading works

A central `GlobalModuleRegistry` manages the lifecycle:

1. **Load:** When a user navigates to a module, the registry creates a new `CollectibleModuleLoadContext(isCollectible: true)`, loads the module's DLL (and its dependencies via `AssemblyDependencyResolver`), and instantiates the module's `IModuleEntry`.

2. **Service registration:** Each module gets its own `IServiceProvider` built from a fresh `ServiceCollection`. The host copies in shared services (`IConfiguration`, `ILoggerFactory`, `IHttpClientFactory`, `IFusionCache`, localization), then calls `moduleEntry.ConfigureServices(services, config)` where the module registers its own services — typically `AddDbContext()`, repositories, business services, `HttpClient` registrations, etc. The resulting `ServiceProvider` is built and stored alongside the ALC.

3. **Rendering:** Modules ship Blazor components (pages, forms, dashboards). The host has a `ModulePage` component that dynamically renders the module's root component inside the host's Blazor circuit. The module's `IServiceProvider` is used as a `CascadingValue` so module components resolve their own services.

4. **Reference counting:** Each active Blazor circuit that renders a module increments a `RefCount`. When the user navigates away, `RefCount` is decremented. Multiple circuits can use the same module version simultaneously.

5. **Unload:** When `RefCount` reaches 0, a grace period timer starts (30 minutes in production, shorter in debug). If no circuit re-acquires the module during the grace period, the registry:
- Disposes the module's `IServiceProvider` (via `IAsyncDisposable`)
- Calls `Alc.Unload()`
- Sets `IsUnloaded = true`
- A background cleanup task later removes the entry from the dictionary

### Multi-tenancy and versioning

It's a multi-tenant system — each tenant has their own database, and module versions are tied to tenant database schema versions. When we ship a patch, we deploy a new version folder on disk. Users on the old schema version keep using the old module version; users who've migrated get the new one. So at any given time, **multiple versions of the same module coexist in memory** (e.g., v1.2.0 and v1.3.0 of the Chat module, each in their own ALC with their own `IServiceProvider`).

We expect ~200 modules active simultaneously with up to 20-50 version bumps/day across them. Reclaiming memory from old module versions is the entire reason we chose collectible ALCs.

## The problem

After the unload sequence completes (`IServiceProvider` disposed, `Unload()` called, all references nulled), the ALC is **never collected** by the GC — even after:

- `GC.Collect(GC.MaxGeneration, GCCollectionMode.Aggressive, blocking: true)` + `WaitForPendingFinalizers()` repeated multiple times
- `WeakReference(alc, trackResurrection: true).IsAlive` stays `true` indefinitely

## What we found following the debugging guide

### Step 1: `dumpheap -type LoaderAllocator`

Fresh process. Loaded a module, used it (Blazor page rendered, EF Core queries ran, JS interop calls made, some hubs actions were invoked + some fusionCache method calls), navigated away, waited for grace period, module unloaded. Loaded the same module again into a new ALC, same usage flow, unloaded again. Triggered aggressive GC. Both `WeakReference.IsAlive` still `true`.

```
> dumpheap -type LoaderAllocator

02195b97e640 48 System.Reflection.LoaderAllocator (ALC _id=1)
02195c624680 48 System.Reflection.LoaderAllocator (ALC _id=2)
```

Two LoaderAllocator objects — one per ALC, confirmed with `dumpobj` (separate addresses, `_isCollectible=true`, `_state=1`).

### Step 2: `gcroot` on the ALC objects

```
> gcroot 02195b97e5c8 (the ALC object itself, _id=1)
HandleTable:
00000219044d4120 (strong handle)
-> 02195b97e5c8 CollectibleModuleLoadContext
Found 1 unique roots.
```

Our code holds **zero** references to the ALC. The single root is the runtime's internal strong handle that exists while the LoaderAllocator is alive.

### Step 3: `gcroot` on LoaderAllocator objects

As the guide recommends, we ran `gcroot` on the LoaderAllocator itself:

```
> gcroot 02195b97e640 (LoaderAllocator for ALC _id=1)
Found 5,686 unique roots.

> gcroot 02195c624680 (LoaderAllocator for ALC _id=2)
Found 5,699 unique roots.
```

The output is a 48MB file with thousands of chains. Here's what we observe:

**Most roots are `(10)` handles.** Out of ~5,690 roots per LoaderAllocator, approximately 5,670 are `HandleTable: (10)` entries. These appear to be dependent handles linking `RuntimeType` objects to the LoaderAllocator. Many are short 2-hop chains:

```
HandleTable:
00000219044d1200 (10)
-> 02195c52a2c0 System.RuntimeType
-> 02195b97e640 System.Reflection.LoaderAllocator
```

**Longer chains go through framework infrastructure.** Some chains are 80-90 hops long, traversing the entire application's object graph. Here's the first chain in the output (abbreviated):

```
HandleTable:
0000021903131600 (10)
-> System.RuntimeType
-> System.Reflection.LoaderAllocator (ALC _id=2) ← passes through the OTHER ALC's LA
-> System.Object[] → ... (HttpClient, TimerQueue, hosted services,
Sentry middleware, ASP.NET routing) ...
-> HubConnectionHandler
-> DefaultHubProtocolResolver
-> JsonHubProtocol
-> System.Text.Json.JsonSerializerOptions ← host's shared instance
-> JsonSerializerOptions+CachingContext
-> ConcurrentDictionary ← Type keys from module ALC
-> ConcurrentDictionary+Node
-> System.RuntimeType ← from module ALC _id=1
-> System.Reflection.LoaderAllocator ← TARGET
```

**The tool annotates some objects as static variables.** Along the chains, `dotnet-dump` marks certain objects with `(static variable: ...)`. We're not sure if these are the actual root causes or just intermediate objects that happen to also be reachable through static fields. Examples from the raw output:

```
000002191763d010 (10)
-> 02195bba07e8 System.Object[]
-> 02195bba3498 Microsoft.Data.SqlClient.MetaType
(static variable: Microsoft.Data.SqlClient.MetaType.s_metaTable)
-> 02195bba34e0 System.RuntimeType
-> 02195b97e640 System.Reflection.LoaderAllocator ← TARGET
```

```
0000021917638be8 (10)
-> 02195ba85a30 System.Object[]
-> 02195ba85bd0 System.Reflection.RuntimeMethodInfo
(static variable: ...ClrPropertySetterFactory.Instance)
-> 02195ba85ac8 System.RuntimeType
-> 02195b97e640 System.Reflection.LoaderAllocator ← TARGET
```

```
000002191763d078 (10)
-> 02195bba2978 System.Object[]
-> 02195bba01c0 System.Action
(static variable: ...SqlDiagnosticListener.s_diagnosticListener)
-> 02195b97e640 System.Reflection.LoaderAllocator ← TARGET
```

### Frameworks appearing in the chains

Based on what we see in the `gcroot` output, the reference chains pass through objects belonging to these frameworks:

- **System.Text.Json** — `JsonSerializerOptions.CachingContext` (via Blazor Server's `JsonHubProtocol`) caches `Type` objects as dictionary keys. Known issue: [dotnet/runtime#65323](https://github.com/dotnet/runtime/issues/65323).
- **EF Core** — `ServiceProviderCache` (static singleton) retaining `DbContextOptions` after module `ServiceProvider` disposal. Various static `MethodInfo`/`PropertyInfo` fields. Related: [dotnet/efcore#31539](https://github.com/dotnet/efcore/issues/31539), [dotnet/efcore#27169](https://github.com/dotnet/efcore/issues/27169).
- **Microsoft.Data.SqlClient** — static caches (`MetaType`, `SqlDiagnosticListener`). Related: [dotnet/SqlClient#1635](https://github.com/dotnet/SqlClient/issues/1635).
- **Microsoft.JSInterop** — `TaskGenericsUtil` caching `Type` objects as dictionary keys.
- **System.Diagnostics** — `ActivitySource`/OpenTelemetry chains.

We can't say with certainty which of these are the definitive blockers vs. intermediate objects in the chains. That's part of why we're filing this.

## Questions

1. **Are we interpreting the `gcroot` output incorrectly, or are we doing something wrong on our side?** We followed the unloadability debugging guide, but the output is complex — thousands of roots, chains up to 90 hops, and `(10)` handle types not covered in the guide. If there's something we're missing or misunderstanding about how to read these results, we'd appreciate guidance.

2. **Which framework caches in the chains are actually responsible for preventing collection?** We see System.Text.Json, EF Core, SqlClient, JSInterop, and diagnostics appearing in the `gcroot` chains, but we can't determine which are true blockers vs. intermediate objects.

3. **Could there ever be an option to forcefully release a collectible ALC?** Even if framework caches hold stale type references, having a way to force the runtime to drop those references and collect the ALC would solve this for us and likely for others in similar scenarios.

## Related issues

We went through several existing issues while looking for answers. Some of these may be related:

- [dotnet/runtime#65323](https://github.com/dotnet/runtime/issues/65323)
- [dotnet/runtime#43544](https://github.com/dotnet/runtime/issues/43544)
- [dotnet/runtime#13283](https://github.com/dotnet/runtime/issues/13283) / [dotnet/runtime#114619](https://github.com/dotnet/runtime/pull/114619)
- [dotnet/runtime#124876](https://github.com/dotnet/runtime/issues/124876)
- [dotnet/runtime#44679](https://github.com/dotnet/runtime/issues/44679)
- [dotnet/runtime#45264](https://github.com/dotnet/runtime/issues/45264)
- [dotnet/SqlClient#1635](https://github.com/dotnet/SqlClient/issues/1635)
- [dotnet/efcore#27169](https://github.com/dotnet/efcore/issues/27169)