Memory leak: Property caches are not getting cleaned up when using AssemblyParts.

### Describe the bug
When you load an assembly in a new AssemblyLoadContext and add it dynamically via an AssemblyPart to the ApplicationPartManager new entries are getting added to the internal property caches. These entries do not get removed when the AssemblyPart is removed and the assembly is unloaded.

These are the caches that i am talking about:
[PropertiesCache](https://github.com/dotnet/aspnetcore/blob/c18fc3d7ca468628c6bbb7030014ab94f15f83cf/src/Shared/PropertyHelper/PropertyHelper.cs#L34)
[VisiblePropertiesCache](https://github.com/dotnet/aspnetcore/blob/c18fc3d7ca468628c6bbb7030014ab94f15f83cf/src/Shared/PropertyHelper/PropertyHelper.cs#L37).

Because same types loaded in a different AssemblyLoadContext are not reference equal more and more entries are getting added.

It would be counterproductive to clear both dictionaries. At least the entries belonging to the unloaded assembly should be removed. Although i guess that the performance impact of clearing the whole cache wouldn't be high.

If you load and unload dlls multiple times you can clearly see the issue.

Without clearing the property caches:

With clearing the property caches:

### To Reproduce
- Load an assembly in an collectible AssemblyLoadContext
- Add the loaded assembly to the ApplicationParts list in the ApplicationPartManager
- Trigger a change via a registered IActionDescriptorChangeProvider
- Remove the previously added AssemblyPart from the ApplicationParts list
- Trigger a change via a registered IActionDescriptorChangeProvider
- Repeat above steps and watch memory consumption

I added a repo that can be used for investigation: [Link](https://github.com/mrukas/PropertyCache-Test)
If line [111 in AssemblyController](https://github.com/mrukas/PropertyCache-Test/blob/master/PropertyCache%20Test/Controllers/AssemblyController.cs#L111) is commented you experience a memory leak. If you trigger [this](https://github.com/mrukas/PropertyCache-Test/blob/master/PropertyCache%20Test/Controllers/AssemblyController.cs#L44) method multiple times you can see in the returned result, that for every load an entry gets added to the PropertiesCache. If you comment [this](https://github.com/mrukas/PropertyCache-Test/blob/master/PropertyCache%20Test/Controllers/AssemblyController.cs#L71) line and call [this](https://github.com/mrukas/PropertyCache-Test/blob/master/PropertyCache%20Test/Controllers/AssemblyController.cs#L64) method in a production build you can see a significant increase in memory consumption. I did not experience an increase of the number of entries in the VisiblePropertiesCache, but I'm pretty sure both caches should be cleaned up if they contain an entry for an unloaded assembly.

### Further technical details
- ASP.NET Core version: 3.1.1
- The IDE (VS / VS Code/ VS4Mac) you're running on, and it's version: Microsoft Visual Studio Professional 2019 Preview
Version 16.5.0 Preview 1.0

> .NET Core SDK (gemäß "global.json"):
> Version: 3.1.100
> Commit: cd82f021f4
>
> Laufzeitumgebung:
> OS Name: Windows
> OS Version: 10.0.18363
> OS Platform: Windows
> RID: win10-x64
> Base Path: C:\Program Files\dotnet\sdk\3.1.100\
>
> Host (useful for support):
> Version: 3.1.1
> Commit: a1388f194c
>
> .NET Core SDKs installed:
> 1.0.4 [C:\Program Files\dotnet\sdk]
> 1.1.0 [C:\Program Files\dotnet\sdk]
> 2.1.2 [C:\Program Files\dotnet\sdk]
> 2.1.4 [C:\Program Files\dotnet\sdk]
> 2.1.101 [C:\Program Files\dotnet\sdk]
> 2.1.104 [C:\Program Files\dotnet\sdk]
> 2.1.105 [C:\Program Files\dotnet\sdk]
> 2.1.201 [C:\Program Files\dotnet\sdk]
> 2.1.202 [C:\Program Files\dotnet\sdk]
> 2.1.300 [C:\Program Files\dotnet\sdk]
> 2.1.301 [C:\Program Files\dotnet\sdk]
> 2.1.302 [C:\Program Files\dotnet\sdk]
> 2.1.400 [C:\Program Files\dotnet\sdk]
> 2.1.402 [C:\Program Files\dotnet\sdk]
> 2.1.403 [C:\Program Files\dotnet\sdk]
> 2.1.504 [C:\Program Files\dotnet\sdk]
> 2.2.106 [C:\Program Files\dotnet\sdk]
> 2.2.202 [C:\Program Files\dotnet\sdk]
> 3.1.100-preview3-014645 [C:\Program Files\dotnet\sdk]
> 3.1.100 [C:\Program Files\dotnet\sdk]
>
> .NET Core runtimes installed:
> Microsoft.AspNetCore.All 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
> Microsoft.AspNetCore.All 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
> Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
> Microsoft.AspNetCore.All 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
> Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
> Microsoft.AspNetCore.All 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
> Microsoft.AspNetCore.All 2.1.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
> Microsoft.AspNetCore.All 2.2.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
> Microsoft.AspNetCore.All 2.2.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
> Microsoft.AspNetCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
> Microsoft.AspNetCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
> Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
> Microsoft.AspNetCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
> Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
> Microsoft.AspNetCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
> Microsoft.AspNetCore.App 2.1.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
> Microsoft.AspNetCore.App 2.2.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
> Microsoft.AspNetCore.App 2.2.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
> Microsoft.AspNetCore.App 3.1.0-preview3.19555.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
> Microsoft.AspNetCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
> Microsoft.AspNetCore.App 3.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
> Microsoft.NETCore.App 1.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 1.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 2.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 2.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 2.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 2.1.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 2.2.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 2.2.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 3.1.0-preview3.19553.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.NETCore.App 3.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
> Microsoft.WindowsDesktop.App 3.1.0-preview3.19553.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
> Microsoft.WindowsDesktop.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]