Issue with missing grain reference when retrieving all reminders

Hi,

I have an issue when I'm trying to retrieve all reminders for a grain which are persisted to DynamoDb. I have the following code:

```
if (Persistence.RecordExists)
await Persistence.ClearStateAsync();

var reminders = await GetReminders();

foreach(var reminder in reminders)
{
await UnregisterReminder(reminder);
}

DeactivateOnIdle();
```

And when it tries to get all the reminders (`var reminders = await GetReminders();`) it first logs the following warning:

```
{
"EventId": 102900,
"LogLevel": "Warning",
"Category": "Orleans.Reminders.DynamoDB.DynamoDBReminderTable",
"Message": "Intermediate error reading reminder entry [{:ServiceId -> Amazon.DynamoDBv2.Model.AttributeValue}\n{:GrainReference -> Amazon.DynamoDBv2.Model.AttributeValue}] from table MCC-reminders.",
"Exception": "Amazon.DynamoDBv2.AmazonDynamoDBException: Query condition missed key schema element: GrainReference ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown. at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken) at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext) --- End of inner exception stack trace --- at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionStream(IRequestContext requestContext, IWebResponseData httpErrorResponse, HttpErrorResponseException exception, Stream responseStream) at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionAsync(IExecutionContext executionContext, HttpErrorResponseException exception) at Amazon.Runtime.Internal.ExceptionHandler`1.HandleAsync(IExecutionContext executionContext, Exception exception) at Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync(IExecutionContext executionContext, Exception exception) at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.Signer.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext) at Orleans.Reminders.DynamoDB.DynamoDBStorage.QueryAsync[TResult](String tableName, Dictionary`2 keys, String keyConditionExpression, Func`2 resolver, String indexName, Boolean scanIndexForward, Dictionary`2 lastEvaluatedKey, Boolean consistentRead) in /_/src/AWS/Shared/Storage/DynamoDBStorage.cs:line 683 at Orleans.Reminders.DynamoDB.DynamoDBStorage.QueryAllAsync[TResult](String tableName, Dictionary`2 keys, String keyConditionExpression, Func`2 resolver, String indexName, Boolean scanIndexForward, Boolean consistentRead) at Orleans.Reminders.DynamoDB.DynamoDBReminderTable.ReadRows(GrainReference grainRef) in /_/src/AWS/Orleans.Reminders.DynamoDB/Reminders/DynamoDBReminderTable.cs:line 148"
}
```

And then throws the following exception:

```
{
"EventId": 0,
"LogLevel": "Error",
"Category": "Infrastructure.GrainExceptionFilter",
"Message": "Query condition missed key schema element: GrainReference ",
"Exception": "Amazon.DynamoDBv2.AmazonDynamoDBException: Query condition missed key schema element: GrainReference ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown. at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken) at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext) --- End of inner exception stack trace --- at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionStream(IRequestContext requestContext, IWebResponseData httpErrorResponse, HttpErrorResponseException exception, Stream responseStream) at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionAsync(IExecutionContext executionContext, HttpErrorResponseException exception) at Amazon.Runtime.Internal.ExceptionHandler`1.HandleAsync(IExecutionContext executionContext, Exception exception) at Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync(IExecutionContext executionContext, Exception exception) at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.Signer.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext) at Orleans.Reminders.DynamoDB.DynamoDBStorage.QueryAsync[TResult](String tableName, Dictionary`2 keys, String keyConditionExpression, Func`2 resolver, String indexName, Boolean scanIndexForward, Dictionary`2 lastEvaluatedKey, Boolean consistentRead) in /_/src/AWS/Shared/Storage/DynamoDBStorage.cs:line 683 at Orleans.Reminders.DynamoDB.DynamoDBStorage.QueryAllAsync[TResult](String tableName, Dictionary`2 keys, String keyConditionExpression, Func`2 resolver, String indexName, Boolean scanIndexForward, Boolean consistentRead) at Orleans.Reminders.DynamoDB.DynamoDBReminderTable.ReadRows(GrainReference grainRef) in /_/src/AWS/Orleans.Reminders.DynamoDB/Reminders/DynamoDBReminderTable.cs:line 148 at Orleans.Runtime.ReminderService.LocalReminderService.GetReminders(GrainReference grainRef) in /_/src/Orleans.Runtime/ReminderService/LocalReminderService.cs:line 165 at Orleans.OrleansCodeGenReminderServiceMethodInvoker.Invoke(IAddressable grain, InvokeMethodRequest request) in /_/src/Orleans.Core/obj/Release/net6.0/Orleans.Core.orleans.g.cs:line 5058 at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 104 at Infrastructure.GrainExceptionFilter.Invoke(IIncomingGrainCallContext context) in /src/services/MonitorAndControlService/src/MonitorAndControlGrains/Infrastructure/GrainExceptionFilter.cs:line 104"
}
```

Might be related, but I'll chuck it in just in case. I also have a grain exception filter where I log the method that failed and the parameters passed to it to help with troubleshooting. I do this using the following `context.Arguments.Where(x => x is not Type).Select(x => JsonSerializer.Serialize(x, options))` where one of the parameters I assume is a reference to the grain that causes the exceptions above. This generates the following exception:

```
{
"EventId": 0,
"LogLevel": "Error",
"Category": "Infrastructure.GrainExceptionFilter",
"Message": "Failed to log error.",
"Exception": "System.InvalidOperationException: This overload of UniqueKey.PrimaryKeyToGuid cannot be used if the grain uses the primary key extension feature. at Orleans.Runtime.UniqueKey.ThrowIfHasKeyExt(String methodName) in /_/src/Orleans.Core.Abstractions/IDs/UniqueKey.cs:line 195 at Orleans.Runtime.GrainId.get_PrimaryKey() in /_/src/Orleans.Core.Abstractions/IDs/GrainId.cs:line 106 at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter`1.WriteCoreAsObject(Utf8JsonWriter writer, Object value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.JsonSerializer.WriteCore[TValue](Utf8JsonWriter writer, TValue& value, JsonTypeInfo`1 jsonTypeInfo) at System.Text.Json.JsonSerializer.WriteString[TValue](TValue& value, JsonTypeInfo`1 jsonTypeInfo) at Infrastructure.GrainExceptionFilter.<>c__DisplayClass3_0.b__1(Object x) in /src/services/MonitorAndControlService/src/MonitorAndControlGrains/Infrastructure/GrainExceptionFilter.cs:line 68 at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext() at System.String.Join(String separator, IEnumerable`1 values) at Infrastructure.GrainExceptionFilter.Invoke(IIncomingGrainCallContext context) in /src/services/MonitorAndControlService/src/MonitorAndControlGrains/Infrastructure/GrainExceptionFilter.cs:line 104"
}
```

So I can see that the grain reference isn't being passed along to the retrieval from DynamoDb. It looks like the grain itself has lost it's own reference at some point - what could be causing this?

Thanks

Trace GetReminders through Orleans.Runtime.ReminderService.LocalReminderService.cs:165 to Orleans.Reminders.DynamoDB.DynamoDBReminderTable.cs:148 and DynamoDBStorage.cs:683. Compare the query keys and key condition with the DynamoDB reminder-table schema, then inspect the grain-reference serialization path implicated by GrainId.PrimaryKey. Done means the missing GrainReference error is resolved and exception logging no longer fails while reporting it.

Written by the indexing model from the issue text.