6.0 dotnet/runtime: Microsoft.DotNet.Compatibility.ValidatePackage fails, "System.MissingMethodException: Method not found: '...'"

I hit this in dotnet/runtime in the online tarball build on 2021-10-11. The conditions aren't clear, but so far it seems like flakiness:

```
/home/dagood/sb/6.0-tb-msbuild/src/runtime.b1aedd5a54d0c54ec12afe98a41979fb053998a5/artifacts/source-build/self/package-cache/microsoft.dotnet.compatibility/1.0.0-rc.2.21419.17/build/Microsoft.NET.Compatibility.Common.targets(26,5):
error MSB4018: The "Microsoft.DotNet.Compatibility.ValidatePackage" task failed unexpectedly.
System.MissingMethodException: Method not found: 'Int32 Microsoft.CodeAnalysis.ISymbol.get_MetadataToken()'.
at Microsoft.CodeAnalysis.CSharp.Symbols.AssemblySymbol.CreateISymbol()
at Microsoft.CodeAnalysis.CSharp.Symbol.get_ISymbol()
at Microsoft.CodeAnalysis.CSharp.Symbols.SymbolExtensions.GetPublicSymbol[TISymbol](Symbol symbol)
at Microsoft.CodeAnalysis.CSharp.Symbols.SymbolExtensions.GetPublicSymbol(Symbol symbol)
at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.CommonGetAssemblyOrModuleSymbol(MetadataReference reference)
at Microsoft.CodeAnalysis.Compilation.GetAssemblyOrModuleSymbol(MetadataReference reference)
at Microsoft.DotNet.ApiCompatibility.AssemblySymbolLoader.LoadAssembly(String name, Stream stream) in /_/src/Compatibility/Microsoft.DotNet.ApiCompatibility/AssemblySymbolLoader.cs:line 166
at Microsoft.DotNet.PackageValidation.ApiCompatRunner.RunApiCompat() in /_/src/Compatibility/Microsoft.DotNet.PackageValidation/ApiCompatRunner.cs:line 47
at Microsoft.DotNet.PackageValidation.CompatibleFrameworkInPackageValidator.Validate(Package package) in /_/src/Compatibility/Microsoft.DotNet.PackageValidation/CompatibleFrameworkInPackageValidator.cs:line 62
at Microsoft.DotNet.Compatibility.ValidatePackage.ExecuteCore() in /_/src/Compatibility/Microsoft.DotNet.Compatibility/ValidatePackage.cs:line 65
at Microsoft.NET.Build.Tasks.TaskBase.Execute() in /_/src/Tasks/Common/TaskBase.cs:line 38
at Microsoft.DotNet.Compatibility.ValidatePackage.Execute() in /_/src/Compatibility/Microsoft.DotNet.Compatibility/ValidatePackage.cs:line 45
at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/home/dagood/sb/6.0-tb-msbuild/src/runtime.b1aedd5a54d0c54ec12afe98a41979fb053998a5/artifacts/source-build/self/src/src/libraries/Microsoft.Extensions.Caching.Abstractions/src/Microsoft.Extensions.Caching.Abstractions.csproj]
```

After I hit the error, I reran `./build.sh --online` in the tarball, and it passed on the second attempt.

Here's the task's subtree in the binlog, with a diff vs. the same tree in a "good" build: https://gist.github.com/dagood/b2bfb711265cd2b7138d740f2060877c

The only difference except the error itself is that the "bad" version has this line:
` NoWarn = ;1701;1702;1705;1591,CS8969;NU5104;NU5104;NU5105`
And the "good" version has this line... just repeating `NU5104` two more times:
` NoWarn = ;1701;1702;1705;1591,CS8969;NU5104;NU5104;NU5104;NU5104;NU5105`

This suggests to me there must be some subtle MSBuild evaluation difference. There are some differences in the binlogs--search for `$project Microsoft.Extensions.Caching.Abstractions.csproj Build netstandard2.0 net461`, pick the second one (the `src/` project, not `ref/`), and go to the evaluation node (click the `id:1264` / `id:1255` link on the project node), copy the subtree, and diff:
https://gist.github.com/dagood/e4abb7b2bd55ce2b0519e3978c9909e3
(Included lots of diff context lines.)

I'm not sure what the significance is, though. (GitHub isn't letting me upload files right now, so I'll try to attach the binlogs later.)

My build was based on dotnet/installer@18d694a877.

---

@eerhardt also hit this (on 2021-10-12):

> I didn't change what SHA I was on for any repos. I just made some .csproj changes in the runtime, ./build.sh --online, got the above error, ./build.sh --online again and was successful
> for installer, I'm on src/installer.e0343e851a6dd66bbae5abc2509a446c02847e5d

---

I searched for the task name in dotnet/source-build and I see that @omajid hit this before, here: https://github.com/dotnet/source-build/issues/2477#issuecomment-927380510 (2021-09-26). I don't see any diagnosis in the thread though--I guess it wasn't related to the main topic in the end:

> I tried replacing `runtime.linux-x64.microsoft.netcore.ilasm` and `runtime.linux-x64.microsoft.netcore.ildasm` with versions downloaded from https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet6/nuget/v3/index.json but it leads to errors in the build:
>
> ```
> dotnet-28be3e9a006d90d8c6e87d4353b77882829df718-x64-bootstrap/src/runtime.826f81a11ad17f415668fe1cb934bdaf00d36ea2/artifacts/source-build/self/package-cache/microsoft.dotnet.compatibility/1.0.0-rc.2.21419.17/build/Microsoft.NET.Compatibility.Common.targets(26,5): error MSB4018: The "Microsoft.DotNet.Compatibility.ValidatePackage" task failed unexpectedly. [dotnet-28be3e9a006d90d8c6e87d4353b77882829df718-x64-bootstrap/src/runtime.826f81a11ad17f415668fe1cb934bdaf00d36ea2/artifacts/source-build/self/src/src/libraries/System.IO.Hashing/src/System.IO.Hashing.csproj]
> dotnet-28be3e9a006d90d8c6e87d4353b77882829df718-x64-bootstrap/src/runtime.826f81a11ad17f415668fe1cb934bdaf00d36ea2/artifacts/source-build/self/package-cache/microsoft.dotnet.compatibility/1.0.0-rc.2.21419.17/build/Microsoft.NET.Compatibility.Common.targets(26,5): error MSB4018: System.MissingMethodException: Method not found: 'Int32 Microsoft.CodeAnalysis.ISymbol.get_MetadataToken()'. [dotnet-28be3e9a006d90d8c6e87d4353b77882829df718-x64-bootstrap/src/runtime.826f81a11ad17f415668fe1cb934bdaf00d36ea2/artifacts/source-build/self/src/src/libraries/System.IO.Hashing/src/System.IO.Hashing.csproj]
> ```
>
> Is this a known issue? Any idea how to work around this?

---

Based on the random hits and retry seeming to always work when attempted, it seems like flakiness. The source of flakiness isn't clear though... it doesn't make sense for this build task to be nondeterministic, and the inputs *appear* effectively identical in the binlog.