NullReferenceException in OutsideRuntimeClient

We've got an Orleans Silo running in Azure in a Linux container app, with a client Azure function app that's pulling from a queue to process data. This runs fine if messages are processed one at a time, but when multiple messages are processed at the same time and the function app is trying to send messages to the same grain concurrently, we get the following exception:

Exception while executing function: Functions.IngestPricesTrigger Result: Failure
Exception: System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
---> System.NullReferenceException: Object reference not set to an instance of an object.
at Orleans.OutsideRuntimeClient.SendRequestMessage(GrainReference target, Message message, IResponseCompletionSource context, InvokeMethodOptions options) in /_/src/Orleans.Core/Runtime/OutsideRuntimeClient.cs:line 244
at Orleans.OutsideRuntimeClient.SendRequest(GrainReference target, IInvokable request, IResponseCompletionSource context, InvokeMethodOptions options) in /_/src/Orleans.Core/Runtime/OutsideRuntimeClient.cs:line 236
at Orleans.Runtime.GrainReferenceRuntime.InvokeMethodAsync[TResult](GrainReference reference, IInvokable request, InvokeMethodOptions options) in /_/src/Orleans.Core/Runtime/GrainReferenceRuntime.cs:line 45
at Orleans.Runtime.GrainReference.InvokeAsync[T](IInvokable methodDescription) in /_/src/Orleans.Core.Abstractions/Runtime/GrainReference.cs:line 413
at OrleansCodeGen.Engine.GrainInterfaces.Proxy_IPriceSourceValidationGrain.global::Engine.GrainInterfaces.IPriceSourceValidationGrain.GetPriceSourceValidationRules() in C:\agents\03\_work\4\s\Engine.GrainInterfaces\Orleans.CodeGenerator\Orleans.CodeGenerator.OrleansSerializationSourceGenerator\Engine.GrainInterfaces.orleans.g.cs:line 415
at Engine.Data.Orleans.Repositories.OrleansPriceSourceRepository.GetPriceSourceValidationRules(Int32 priceSourceId) in C:\agents\03\_work\4\s\Engine.Data.Orleans\Repositories\OrleansPriceSourceRepository.cs:line 44
at Engine.Logic.Implementations.PriceValidationService.ValidateUnprocessedPrices(UnprocessedPrice[] prices, Int32 priceSourceId) in C:\agents\03\_work\4\s\Engine.Logic\Implementations\PriceValidationService.cs:line 28
at Engine.Logic.Implementations.PricingService.ProcessBatchPrices(UnprocessedPrice[] unprocessedPrices, Int32 priceSourceId, DateTime receviedDate, Nullable`1 reportedDate, String fileName, Nullable`1 userId, Guid processGuid) in C:\agents\03\_work\4\s\Engine.Logic\Implementations\PricingService.cs:line 40
at Engine.Functions.Triggers.IngestPricesTrigger.Run(IngestPriceBatchCommand payload) in C:\agents\03\_work\4\s\Engine.Functions\Triggers\IngestPricesTrigger.cs:line 26
at Microsoft.Azure.Functions.Worker.Invocation.VoidTaskMethodInvoker`2.InvokeAsync(TReflected instance, Object[] arguments) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\VoidTaskMethodInvoker.cs:line 22
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task`1.get_Result()
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__273_0(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 44
at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
at Microsoft.Azure.Functions.Worker.GrpcWorker.InvocationRequestHandlerAsync(InvocationRequest request, IFunctionsApplication application, IInvocationFeaturesFactory invocationFeaturesFactory, ObjectSerializer serializer, IOutputBindingsInfoProvider outputBindingsInfoProvider, IInputConversionFeatureProvider functionInputConversionFeatureProvider) in D:\a\_work\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 199
Stack: at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task`1.get_Result()
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__273_0(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 44
at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
at Microsoft.Azure.Functions.Worker.GrpcWorker.InvocationRequestHandlerAsync(InvocationRequest request, IFunctionsApplication application, IInvocationFeaturesFactory invocationFeaturesFactory, ObjectSerializer serializer, IOutputBindingsInfoProvider outputBindingsInfoProvider, IInputConversionFeatureProvider functionInputConversionFeatureProvider) in D:\a\_work\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 199

This is the code that's used to get the grain. PriceSourceId is never going to be null, so that's not the issue.

```
var priceSourceValidation = _orleansClusterClient.GetGrain(priceSourceId);
return await priceSourceValidation.GetPriceSourceValidationRules();
```

We're using Azure table storage for the membership table.

Start in src/Orleans.Core/Runtime/OutsideRuntimeClient.cs around line 244, then trace the request path shown through GrainReferenceRuntime and GrainReference. Reproduce concurrent calls from the Azure client against the same grain and inspect which runtime state is null. Done means concurrent requests no longer produce a NullReferenceException; the payload does not name a regression test.

Written by the indexing model from the issue text.