I have a weird issue with Orleans, that I didn't have before, using Consul as membership provider

Hi,

I'm using a fairly simple configuration, with docker images and Consul as membership provider for my setup. The configuration is with 2 silos, one grain per silo (as a test bench). The issue, that I have not observed in earlier versions of Orleans, is that there is always a startup problem, when I run the setup in Visual Studio as a debug session.

One or the other grain can't be resolved by the grain factory, giving the following message

`Could not find an implementation for interface`

E.g., in one startup we have the message

`Could not find an implementation for interface ITestGrain001`

And then in another startup

`Could not find an implementation for interface ITestGrain002`

It's random.

I've put the factory (resolving the grain) into a try-catch loop, and then clocked in that loop how long time it takes for the factory to be able to resolve the grain, and it takes one minute. Then the error goes away and mostly everything works fine.

But sometimes it gets even worse.

We have another issue, once in awhile when the factory successfully resolves the grain (after the one minute retry):

`System.IO.FileNotFoundException: Could not load file or assembly`

This exception targets the grain implementation, for the grain interface.

I've also put the invocation to the grain into a try-catch loop, and it turns out that after the first try and fail (loading the assembly), the second has always succeeded.

After doing these retrials, and everything seems loaded, the setup runs fine. Until I bring everything down, and up again, and then we're back to this behavior.

Here is the stack trace when the interface can't be resolved (in this case for ITestGrain001):

System.ArgumentException: Could not find an implementation for interface TestBench.ITestGrain001
at Orleans.GrainInterfaceTypeToGrainTypeResolver.GetGrainType(GrainInterfaceType interfaceType) in /_/src/Orleans.Core/Core/GrainInterfaceTypeToGrainTypeResolver.cs:line 102
at Orleans.GrainFactory.GetGrain[TGrainInterface](Guid primaryKey, String grainClassNamePrefix) in /_/src/Orleans.Core/Core/GrainFactory.cs:line 45

Here is the stack trace when the assembly can't load:

at System.Reflection.RuntimeAssembly.g____PInvoke|47_0(NativeAssemblyNameParts* pAssemblyNameParts, ObjectHandleOnStack requestingAssembly, StackCrawlMarkHandle stackMark, Int32 throwOnFileNotFound, ObjectHandleOnStack assemblyLoadContext, ObjectHandleOnStack retAssembly)
at System.Reflection.RuntimeAssembly.InternalLoad(AssemblyName assemblyName, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext, RuntimeAssembly requestingAssembly, Boolean throwOnFileNotFound)
at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
at Orleans.Serialization.TypeSystem.CachedTypeResolver.g__ResolveAssembly|9_0(AssemblyName assemblyName) in /_/src/Orleans.Serialization/TypeSystem/CachedTypeResolver.cs:line 141
at System.TypeNameParser.ResolveAssembly(String asmName, Func`2 assemblyResolver, Boolean throwOnError, StackCrawlMark& stackMark)
at System.TypeNameParser.ConstructType(Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark)
at System.TypeNameParser.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark)
at System.Type.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError)
at Orleans.Serialization.TypeSystem.CachedTypeResolver.TryPerformUncachedTypeResolution(String fullName, Type& type, Assembly[] assemblies) in /_/src/Orleans.Serialization/TypeSystem/CachedTypeResolver.cs:line 113
at Orleans.Serialization.TypeSystem.CachedTypeResolver.TryPerformUncachedTypeResolution(String name, Type& type) in /_/src/Orleans.Serialization/TypeSystem/CachedTypeResolver.cs:line 57
at Orleans.Serialization.TypeSystem.CachedTypeResolver.TryResolveType(String name, Type& type) in /_/src/Orleans.Serialization/TypeSystem/CachedTypeResolver.cs:line 45
at Orleans.Serialization.TypeSystem.TypeConverter.ParseInternal(TypeSpec parsed, Type& type) in /_/src/Orleans.Serialization/TypeSystem/TypeConverter.cs:line 332
at Orleans.Serialization.TypeSystem.TypeConverter.ResolveCompoundAliasType[TState](TupleTypeSpec input, TState& state) in /_/src/Orleans.Serialization/TypeSystem/TypeConverter.cs:line 555
at Orleans.Serialization.TypeSystem.RuntimeTypeNameRewriter.TypeRewriter`1.HandleCompoundType(TupleTypeSpec type, String assemblyName) in /_/src/Orleans.Serialization/TypeSystem/RuntimeTypeNameRewriter.cs:line 233
at Orleans.Serialization.TypeSystem.RuntimeTypeNameRewriter.TypeRewriter`1.ApplyInner(TypeSpec input, String assemblyName) in /_/src/Orleans.Serialization/TypeSystem/RuntimeTypeNameRewriter.cs:line 77
at Orleans.Serialization.TypeSystem.RuntimeTypeNameRewriter.TypeRewriter`1.Rewrite(TypeSpec input) in /_/src/Orleans.Serialization/TypeSystem/RuntimeTypeNameRewriter.cs:line 72
at Orleans.Serialization.TypeSystem.RuntimeTypeNameRewriter.Rewrite[TState](TypeSpec input, Rewriter`1 rewriter, CompoundAliasResolver`1 compoundAliasRewriter, TState& state) in /_/src/Orleans.Serialization/TypeSystem/RuntimeTypeNameRewriter.cs:line 45
at Orleans.Serialization.TypeSystem.TypeConverter.ParseInternal(TypeSpec parsed, Type& type) in /_/src/Orleans.Serialization/TypeSystem/TypeConverter.cs:line 316
at Orleans.Serialization.TypeSystem.TypeCodec.TryRead[TInput](Reader`1& reader) in /_/src/Orleans.Serialization/TypeSystem/TypeCodec.cs:line 72
at Orleans.Serialization.Codecs.FieldHeaderCodec.ReadType[TInput](Reader`1& reader, SchemaType schemaType) in /_/src/Orleans.Serialization/Codecs/FieldHeaderCodec.cs:line 187
at Orleans.Serialization.Codecs.FieldHeaderCodec.ReadExtendedFieldHeader[TInput](Reader`1& reader, Field& field) in /_/src/Orleans.Serialization/Codecs/FieldHeaderCodec.cs:line 169
at Orleans.Runtime.Messaging.MessageSerializer.ReadBodyObject[TInput](Message message, Reader`1& reader) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 130
at Orleans.Runtime.Messaging.MessageSerializer.TryRead(ReadOnlySequence`1& input, Message& message) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 126
at Orleans.Runtime.Messaging.Connection.ProcessIncoming() in /_/src/Orleans.Core/Networking/Connection.cs:line 346

Also, the documentation about using [Consul as a membership provider](https://learn.microsoft.com/en-us/dotnet/orleans/deployment/consul-deployment) is sadly outdated. There are changes made to how to use Consul in the code, but this is not reflected at all by the documentation.

Thanks