RouteValueDictionary prevents assembly from being unloaded

I am trying to load and unload an assembly containing controllers using the way explained here:
https://docs.microsoft.com/en-us/dotnet/standard/assembly/unloadability

Having trouble unloading the assembly and having a look with WinDbg, I found that RouteValueDictionary, probably used by the endpoint middleware is holding references to types in the assembly, which seems to prevent it from being unloaded.

`0000027274F313E0 (pinned handle)
-> 00000275A5223050 System.Object[]
-> 000002747526DD00 System.Collections.Concurrent.ConcurrentDictionary``2[[System.Type, System.Private.CoreLib],[Microsoft.Extensions.Internal.PropertyHelper[], Microsoft.AspNetCore.Mvc.Core]]
-> 000002747526E018 System.Collections.Concurrent.ConcurrentDictionary``2+Tables[[System.Type, System.Private.CoreLib],[Microsoft.Extensions.Internal.PropertyHelper[], Microsoft.AspNetCore.Mvc.Core]]
-> 000002747526DF08 System.Collections.Concurrent.ConcurrentDictionary``2+Node[[System.Type, System.Private.CoreLib],[Microsoft.Extensions.Internal.PropertyHelper[], Microsoft.AspNetCore.Mvc.Core]][]
-> 00000274752709B0 System.Collections.Concurrent.ConcurrentDictionary``2+Node[[System.Type, System.Private.CoreLib],[Microsoft.Extensions.Internal.PropertyHelper[], Microsoft.AspNetCore.Mvc.Core]]
-> 000002747526B968 System.RuntimeType
-> 000002747526B2C0 System.Reflection.LoaderAllocator`

https://github.com/aspnet/AspNetCore/blob/425c196cba530b161b120a57af8f1dd513b96f67/src/Http/Http.Abstractions/src/Routing/RouteValueDictionary.cs#L726

Is there a way to remove this reference or am I stuck with the assembly in memory?