dotnet / dotnet/orleans

Cluster Manifest Broken by v7.2.5

Open
#8,910 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
14h 42m
Merged PRs (30d)
354

Description

We experienced an issue where after bumping from `v7.2.4` to `v7.2.6` we were getting from following logs from our remote cluster:

> Exception reading message "Request [ sys.client/97a1e507408e496e91dda84405325844]->[S10.16.5.43:30004:0 sys.svc.manifest/10.16.5.43:30004@0] #1" from remote endpoint "10.16.5.48:52438" to local endpoint "10.16.5.43:30004"

With the following exception:

> System.TypeLoadException: Unable to resolve type alias "("inv",[Orleans.Runtime.GrainReference],[Orleans.Runtime.IClusterManifestSystemTarget,Orleans.Core],"4EFCA109")".
at Orleans.Serialization.TypeSystem.TypeConverter.ResolveCompoundAliasType[TState](TupleTypeSpec input, TState& state) in /_/src/Orleans.Serialization/TypeSystem/TypeConverter.cs:line 574
at Orleans.Serialization.TypeSystem.RuntimeTypeNameRewriter.TypeRewriter`1.HandleCompoundType(TupleTypeSpec type, String assemblyName) in /_/src/Orleans.Serialization/TypeSystem/RuntimeTypeNameRewriter.cs:line 233
at Orleans.Serialization.TypeSystem.RuntimeTypeNameRewriter.TypeRewriter`1.ApplyInner(TypeSpec input, String assemblyName) in /_/src/Orleans.Serialization/TypeSystem/RuntimeTypeNameRewriter.cs:line 77
at Orleans.Serialization.TypeSystem.TypeConverter.ParseInternal(TypeSpec parsed, Type& type) in /_/src/Orleans.Serialization/TypeSystem/TypeConverter.cs:line 345
at Orleans.Serialization.TypeSystem.TypeCodec.TryRead[TInput](Reader`1& reader) in /_/src/Orleans.Serialization/TypeSystem/TypeCodec.cs:line 72
at Orleans.Serialization.Codecs.FieldHeaderCodec.ReadType[TInput](Reader`1& reader, SchemaType schemaType) in /_/src/Orleans.Serialization/Codecs/FieldHeaderCodec.cs:line 187
at Orleans.Serialization.Codecs.FieldHeaderCodec.ReadExtendedFieldHeader[TInput](Reader`1& reader, Field& field) in /_/src/Orleans.Serialization/Codecs/FieldHeaderCodec.cs:line 169
at Orleans.Runtime.Messaging.MessageSerializer.ReadBodyObject[TInput](Message message, Reader`1& reader) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 130
at Orleans.Runtime.Messaging.MessageSerializer.TryRead(ReadOnlySequence`1& input, Message& message) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 126
at Orleans.Runtime.Messaging.Connection.ProcessIncoming() in /_/src/Orleans.Core/Networking/Connection.cs:line 349

Followed by the following log from our client host:

> Failed to fetch cluster manifest update from "GrainReference:sys.svc.manifest/10.16.5.43:30004@0:Orleans.Runtime.IClusterManifestSystemTarget".

These logs occur every minute, on every attempt to retrieve cluster manifest.

We managed to fix this by downgrading all of our Orleans references to `v7.2.4`. We noticed that there was some kind of change related to the cluster manifest in the [v7.2.5 release](https://github.com/dotnet/orleans/releases/tag/v7.2.5) that might be related?

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the v7.2.4-to-v7.2.6 upgrade and trace the cluster-manifest request from Orleans.Runtime.Messaging.MessageSerializer into TypeConverter.ResolveCompoundAliasType. Read TypeConverter.cs, RuntimeTypeNameRewriter.cs, and the v7.2.5 release notes for the manifest change. Done means upgraded hosts can fetch manifest updates without the TypeLoadException.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.