Adding AddHttpMessageHandler<BaseAddressAuthorizationMessageHandler>() to AddHttpClient call results in a "Specified cast is not valid" error

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Describe the bug

Without this code, my API calls for page refreshes or deep linking on pages results in unauthenticated API requests. When I try and add it, everything will compile fine but at runtime in the browser console I see the following error (after removing all the web assembly gunk.

`
ManagedError: One or more errors occurred. (Specified cast is not valid.)
Error deserializing root component operations: System.InvalidOperationException: No root component exists with SSR component ID 2.
at Microsoft.AspNetCore.Components.WebAssembly.Rendering.WebAssemblyRenderer.WebRootComponentManager.GetRequiredWebRootComponent(Int32 ssrComponentId)
at Microsoft.AspNetCore.Components.WebAssembly.Rendering.WebAssemblyRenderer.WebRootComponentManager.RemoveRootComponent(Int32 ssrComponentId)
at Microsoft.AspNetCore.Components.WebAssembly.Rendering.WebAssemblyRenderer.OnUpdateRootComponents(RootComponentOperationBatch batch)
at Microsoft.AspNetCore.Components.WebAssembly.Services.DefaultWebAssemblyJSRuntime.UpdateRootComponentsCore(String operationsJson)
`

### Expected Behavior

Either for this to work or for me to understand the error better so I can resolve it and understand why adding this http handler results in my application breaking.

### Steps To Reproduce

Code in Blazor wasm client project. LeaguesClient is a class that takes an HttpClient as a parameter and saves it for use to make API calls to a Blazor server API project.

```cs
builder.Services.AddHttpClient("ServerAPI",
client =>
{
client.BaseAddress = new Uri(blazorServerUri);
client.DefaultRequestHeaders.Add("User-Agent", "JustPickem.Client");
}).AddHttpMessageHandler();

builder.Services.AddSingleton(sp => new LeaguesClient(sp.GetRequiredService().CreateClient("ServerAPI")));
```

### Exceptions (if any)

Full raw exception

```
blazor.web.js:1 ManagedError: One or more errors occurred. (Specified cast is not valid.)
callEntryPoint @ blazor.web.js:1
await in callEntryPoint
Hr @ blazor.web.js:1
await in Hr
Fr @ blazor.web.js:1
startWebAssemblyIfNotStarted @ blazor.web.js:1
resolveRendererIdForDescriptor @ blazor.web.js:1
determinePendingOperation @ blazor.web.js:1
refreshRootComponents @ blazor.web.js:1
(anonymous) @ blazor.web.js:1
setTimeout
rootComponentsMayRequireRefresh @ blazor.web.js:1
onDocumentUpdated @ blazor.web.js:1
Hi @ blazor.web.js:1
[NEW] Explain Console errors by using Copilot in Edge: click

to explain an error.
Learn more
Don't show again
blazor.web.js:1 Error deserializing root component operations: System.InvalidOperationException: No root component exists with SSR component ID 2.
gr @ blazor.web.js:1
put_char @ dotnet.native.st0ovrfdhi.js:8
write @ dotnet.native.st0ovrfdhi.js:8
write @ dotnet.native.st0ovrfdhi.js:8
doWritev @ dotnet.native.st0ovrfdhi.js:8
_fd_write @ dotnet.native.st0ovrfdhi.js:8
$func8623 @ 00b5b99e:0x212300
$func4003 @ 00b5b99e:0x114a68
$func288 @ 00b5b99e:0x1f228
$func219 @ 00b5b99e:0x1c8c7
$func210 @ 00b5b99e:0xea32
$func247 @ 00b5b99e:0x1d442
$func3222 @ 00b5b99e:0xf1143
$func2545 @ 00b5b99e:0xc6ca7
$func2550 @ 00b5b99e:0xc749f
$func2574 @ 00b5b99e:0xc9b31
$mono_wasm_invoke_jsexport @ 00b5b99e:0xaccb
Module._mono_wasm_invoke_jsexport @ dotnet.native.st0ovrfdhi.js:8
gn @ managed-exports.ts:297
(anonymous) @ invoke-cs.ts:148
Hr.wt._internal.updateRootComponents @ blazor.web.js:1
(anonymous) @ blazor.web.js:1
updateWebAssemblyRootComponents @ blazor.web.js:1
refreshRootComponents @ blazor.web.js:1
(anonymous) @ blazor.web.js:1
setTimeout
rootComponentsMayRequireRefresh @ blazor.web.js:1
onDocumentUpdated @ blazor.web.js:1
documentUpdated @ blazor.web.js:1
(anonymous) @ blazor.web.js:1
write @ blazor.web.js:1
blazor.web.js:1 at Microsoft.AspNetCore.Components.WebAssembly.Rendering.WebAssemblyRenderer.WebRootComponentManager.GetRequiredWebRootComponent(Int32 ssrComponentId)
gr @ blazor.web.js:1
put_char @ dotnet.native.st0ovrfdhi.js:8
write @ dotnet.native.st0ovrfdhi.js:8
write @ dotnet.native.st0ovrfdhi.js:8
doWritev @ dotnet.native.st0ovrfdhi.js:8
_fd_write @ dotnet.native.st0ovrfdhi.js:8
$func8623 @ 00b5b99e:0x212300
$func4003 @ 00b5b99e:0x114a68
$func288 @ 00b5b99e:0x1f228
$func219 @ 00b5b99e:0x1c8c7
$func210 @ 00b5b99e:0xea32
$func247 @ 00b5b99e:0x1d442
$func3222 @ 00b5b99e:0xf1143
$func2545 @ 00b5b99e:0xc6ca7
$func2550 @ 00b5b99e:0xc749f
$func2574 @ 00b5b99e:0xc9b31
$mono_wasm_invoke_jsexport @ 00b5b99e:0xaccb
Module._mono_wasm_invoke_jsexport @ dotnet.native.st0ovrfdhi.js:8
gn @ managed-exports.ts:297
(anonymous) @ invoke-cs.ts:148
Hr.wt._internal.updateRootComponents @ blazor.web.js:1
(anonymous) @ blazor.web.js:1
updateWebAssemblyRootComponents @ blazor.web.js:1
refreshRootComponents @ blazor.web.js:1
(anonymous) @ blazor.web.js:1
setTimeout
rootComponentsMayRequireRefresh @ blazor.web.js:1
onDocumentUpdated @ blazor.web.js:1
documentUpdated @ blazor.web.js:1
(anonymous) @ blazor.web.js:1
write @ blazor.web.js:1
blazor.web.js:1 at Microsoft.AspNetCore.Components.WebAssembly.Rendering.WebAssemblyRenderer.WebRootComponentManager.RemoveRootComponent(Int32 ssrComponentId)
gr @ blazor.web.js:1
put_char @ dotnet.native.st0ovrfdhi.js:8
write @ dotnet.native.st0ovrfdhi.js:8
write @ dotnet.native.st0ovrfdhi.js:8
doWritev @ dotnet.native.st0ovrfdhi.js:8
_fd_write @ dotnet.native.st0ovrfdhi.js:8
$func8623 @ 00b5b99e:0x212300
$func4003 @ 00b5b99e:0x114a68
$func288 @ 00b5b99e:0x1f228
$func219 @ 00b5b99e:0x1c8c7
$func210 @ 00b5b99e:0xea32
$func247 @ 00b5b99e:0x1d442
$func3222 @ 00b5b99e:0xf1143
$func2545 @ 00b5b99e:0xc6ca7
$func2550 @ 00b5b99e:0xc749f
$func2574 @ 00b5b99e:0xc9b31
$mono_wasm_invoke_jsexport @ 00b5b99e:0xaccb
Module._mono_wasm_invoke_jsexport @ dotnet.native.st0ovrfdhi.js:8
gn @ managed-exports.ts:297
(anonymous) @ invoke-cs.ts:148
Hr.wt._internal.updateRootComponents @ blazor.web.js:1
(anonymous) @ blazor.web.js:1
updateWebAssemblyRootComponents @ blazor.web.js:1
refreshRootComponents @ blazor.web.js:1
(anonymous) @ blazor.web.js:1
setTimeout
rootComponentsMayRequireRefresh @ blazor.web.js:1
onDocumentUpdated @ blazor.web.js:1
documentUpdated @ blazor.web.js:1
(anonymous) @ blazor.web.js:1
write @ blazor.web.js:1
blazor.web.js:1 at Microsoft.AspNetCore.Components.WebAssembly.Rendering.WebAssemblyRenderer.OnUpdateRootComponents(RootComponentOperationBatch batch)
gr @ blazor.web.js:1
put_char @ dotnet.native.st0ovrfdhi.js:8
write @ dotnet.native.st0ovrfdhi.js:8
write @ dotnet.native.st0ovrfdhi.js:8
doWritev @ dotnet.native.st0ovrfdhi.js:8
_fd_write @ dotnet.native.st0ovrfdhi.js:8
$func8623 @ 00b5b99e:0x212300
$func4003 @ 00b5b99e:0x114a68
$func288 @ 00b5b99e:0x1f228
$func219 @ 00b5b99e:0x1c8c7
$func210 @ 00b5b99e:0xea32
$func247 @ 00b5b99e:0x1d442
$func3222 @ 00b5b99e:0xf1143
$func2545 @ 00b5b99e:0xc6ca7
$func2550 @ 00b5b99e:0xc749f
$func2574 @ 00b5b99e:0xc9b31
$mono_wasm_invoke_jsexport @ 00b5b99e:0xaccb
Module._mono_wasm_invoke_jsexport @ dotnet.native.st0ovrfdhi.js:8
gn @ managed-exports.ts:297
(anonymous) @ invoke-cs.ts:148
Hr.wt._internal.updateRootComponents @ blazor.web.js:1
(anonymous) @ blazor.web.js:1
updateWebAssemblyRootComponents @ blazor.web.js:1
refreshRootComponents @ blazor.web.js:1
(anonymous) @ blazor.web.js:1
setTimeout
rootComponentsMayRequireRefresh @ blazor.web.js:1
onDocumentUpdated @ blazor.web.js:1
documentUpdated @ blazor.web.js:1
(anonymous) @ blazor.web.js:1
write @ blazor.web.js:1
blazor.web.js:1 at Microsoft.AspNetCore.Components.WebAssembly.Services.DefaultWebAssemblyJSRuntime.UpdateRootComponentsCore(String operationsJson)
gr @ blazor.web.js:1
put_char @ dotnet.native.st0ovrfdhi.js:8
write @ dotnet.native.st0ovrfdhi.js:8
write @ dotnet.native.st0ovrfdhi.js:8
doWritev @ dotnet.native.st0ovrfdhi.js:8
_fd_write @ dotnet.native.st0ovrfdhi.js:8
$func8623 @ 00b5b99e:0x212300
$func4003 @ 00b5b99e:0x114a68
$func288 @ 00b5b99e:0x1f228
$func219 @ 00b5b99e:0x1c8c7
$func210 @ 00b5b99e:0xea32
$func247 @ 00b5b99e:0x1d442
$func3222 @ 00b5b99e:0xf1143
$func2545 @ 00b5b99e:0xc6ca7
$func2550 @ 00b5b99e:0xc749f
$func2574 @ 00b5b99e:0xc9b31
$mono_wasm_invoke_jsexport @ 00b5b99e:0xaccb
Module._mono_wasm_invoke_jsexport @ dotnet.native.st0ovrfdhi.js:8
gn @ managed-exports.ts:297
(anonymous) @ invoke-cs.ts:148
Hr.wt._internal.updateRootComponents @ blazor.web.js:1
(anonymous) @ blazor.web.js:1
updateWebAssemblyRootComponents @ blazor.web.js:1
refreshRootComponents @ blazor.web.js:1
(anonymous) @ blazor.web.js:1
setTimeout
rootComponentsMayRequireRefresh @ blazor.web.js:1
onDocumentUpdated @ blazor.web.js:1
documentUpdated @ blazor.web.js:1
(anonymous) @ blazor.web.js:1
write @ blazor.web.js:1
```

### .NET Version

9.0.302

### Anything else?

IDE: Visual Studio 2022

Dotnet Info:

```
.NET SDK:
Version: 9.0.302
Commit: bb2550b9af
Workload version: 9.0.300-manifests.183aaee6
MSBuild version: 17.14.13+65391c53b

Runtime Environment:
OS Name: Windows
OS Version: 10.0.26100
OS Platform: Windows
RID: win-x64
Base Path: C:\Program Files\dotnet\sdk\9.0.302\

.NET workloads installed:
[aspire]
Installation Source: SDK 9.0.300, VS 17.14.36301.6
Manifest Version: 8.2.2/8.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\8.0.100\microsoft.net.sdk.aspire\8.2.2\WorkloadManifest.json
Install Type: Msi

[wasm-tools]
Installation Source: SDK 9.0.300
Manifest Version: 9.0.7/9.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\9.0.100\microsoft.net.workload.mono.toolchain.current\9.0.7\WorkloadManifest.json
Install Type: Msi

Configured to use loose manifests when installing new manifests.

Host:
Version: 9.0.7
Architecture: x64
Commit: 3c298d9f00

.NET SDKs installed:
3.1.426 [C:\Program Files\dotnet\sdk]
5.0.214 [C:\Program Files\dotnet\sdk]
5.0.408 [C:\Program Files\dotnet\sdk]
6.0.428 [C:\Program Files\dotnet\sdk]
7.0.410 [C:\Program Files\dotnet\sdk]
9.0.100-rc.2.24474.11 [C:\Program Files\dotnet\sdk]
9.0.108 [C:\Program Files\dotnet\sdk]
9.0.302 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 9.0.0-rc.2.24474.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 9.0.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.18 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.0-rc.2.24473.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.18 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 9.0.0-rc.2.24474.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 9.0.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
Not set

global.json file:
Not found
```

Start with the client registration using AddHttpClient("ServerAPI") and AddHttpMessageHandler(), then inspect the BaseAddressAuthorizationMessageHandler and the WebAssemblyRenderer stack shown in the exception. Reproduce with the provided .NET 9 setup, comparing behavior with the handler absent and present. Done means the authorization handler can be added without the specified-cast or root-component errors.

Written by the indexing model from the issue text.