Cluster Manifest Broken by v7.2.5
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 14h 42m
- Merged PRs (30d)
- 354
Description
We experienced an issue where after bumping from `v7.2.4` to `v7.2.6` we were getting from following logs from our remote cluster:
> Exception reading message "Request [ sys.client/97a1e507408e496e91dda84405325844]->[S10.16.5.43:30004:0 sys.svc.manifest/10.16.5.43:30004@0] #1" from remote endpoint "10.16.5.48:52438" to local endpoint "10.16.5.43:30004"
With the following exception:
> System.TypeLoadException: Unable to resolve type alias "("inv",[Orleans.Runtime.GrainReference],[Orleans.Runtime.IClusterManifestSystemTarget,Orleans.Core],"4EFCA109")".
at Orleans.Serialization.TypeSystem.TypeConverter.ResolveCompoundAliasType[TState](TupleTypeSpec input, TState& state) in /_/src/Orleans.Serialization/TypeSystem/TypeConverter.cs:line 574
at Orleans.Serialization.TypeSystem.RuntimeTypeNameRewriter.TypeRewriter`1.HandleCompoundType(TupleTypeSpec type, String assemblyName) in /_/src/Orleans.Serialization/TypeSystem/RuntimeTypeNameRewriter.cs:line 233
at Orleans.Serialization.TypeSystem.RuntimeTypeNameRewriter.TypeRewriter`1.ApplyInner(TypeSpec input, String assemblyName) in /_/src/Orleans.Serialization/TypeSystem/RuntimeTypeNameRewriter.cs:line 77
at Orleans.Serialization.TypeSystem.TypeConverter.ParseInternal(TypeSpec parsed, Type& type) in /_/src/Orleans.Serialization/TypeSystem/TypeConverter.cs:line 345
at Orleans.Serialization.TypeSystem.TypeCodec.TryRead[TInput](Reader`1& reader) in /_/src/Orleans.Serialization/TypeSystem/TypeCodec.cs:line 72
at Orleans.Serialization.Codecs.FieldHeaderCodec.ReadType[TInput](Reader`1& reader, SchemaType schemaType) in /_/src/Orleans.Serialization/Codecs/FieldHeaderCodec.cs:line 187
at Orleans.Serialization.Codecs.FieldHeaderCodec.ReadExtendedFieldHeader[TInput](Reader`1& reader, Field& field) in /_/src/Orleans.Serialization/Codecs/FieldHeaderCodec.cs:line 169
at Orleans.Runtime.Messaging.MessageSerializer.ReadBodyObject[TInput](Message message, Reader`1& reader) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 130
at Orleans.Runtime.Messaging.MessageSerializer.TryRead(ReadOnlySequence`1& input, Message& message) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 126
at Orleans.Runtime.Messaging.Connection.ProcessIncoming() in /_/src/Orleans.Core/Networking/Connection.cs:line 349
Followed by the following log from our client host:
> Failed to fetch cluster manifest update from "GrainReference:sys.svc.manifest/10.16.5.43:30004@0:Orleans.Runtime.IClusterManifestSystemTarget".
These logs occur every minute, on every attempt to retrieve cluster manifest.
We managed to fix this by downgrading all of our Orleans references to `v7.2.4`. We noticed that there was some kind of change related to the cluster manifest in the [v7.2.5 release](https://github.com/dotnet/orleans/releases/tag/v7.2.5) that might be related?
Contributor guide
Research direction
Start by reproducing the v7.2.4-to-v7.2.6 upgrade and trace the cluster-manifest request from Orleans.Runtime.Messaging.MessageSerializer into TypeConverter.ResolveCompoundAliasType. Read TypeConverter.cs, RuntimeTypeNameRewriter.cs, and the v7.2.5 release notes for the manifest change. Done means upgraded hosts can fetch manifest updates without the TypeLoadException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100