Can't load loggers from Runtime package store

### Steps to reproduce

I've somehow installed some msbuild loggers (e.g, https://www.nuget.org/packages/MSBuild.StructuredLogger/) with `dotnet store` command.

Manifest file
```xml

netcoreapp3.1

```

Command line
```
dotnet store --manifest RuntimePackages.manifest --framework netcoreapp3.1 --runtime win10-x64
dotnet build -l:BinaryLogger,StructuredLogger
```
### Expected behavior

Can use `BinaryLogger` in the assembly `StructuredLogger`, which is installed in `~/.dotnet/store/**/`

### Actual behavior
```
.NET Core용 Microsoft (R) Build Engine 버전 16.4.0+e901037fe
Copyright (C) Microsoft Corporation. All rights reserved.

MSBUILD : error MSB1021: 로거의 인스턴스를 만들 수 없습니다. Could not load file or assembly 'GithubActionsDotnetLogger, Culture=neutral, PublicKeyToken=null'. 지정된 파일을 찾을 수 없습니다.
스위치: GithubActionsDotnetLogger
```

### Environment data
`msbuild /version` output:

OS info:

If applicable, version of the tool that invokes MSBuild (Visual Studio, dotnet CLI, etc):