Unmanaged memory keeps increasing if local direct ClusterClient is used.

**Environment:**
- Orleans version：3.4.1
- .NET Core : 3.1
- OS: Windows 2016/2019 server, Ubuntu 18LTS
- Orleans silo configuration: 3 nodes VMs. one silo each node. Consul clustering.

**Issue:**

- We are observing non-stop unmanaged memory growth when a local direct ClusterClient is used to call grains in the local silo, instead of a regular ClusterClient from ClientBuilder. (referenced https://learn.microsoft.com/en-us/dotnet/orleans/host/client#obtain-a-client-from-a-host)

- As the silo process exhausts the system memory, then it suicides and restarts.

- As shown in the screenshot, we performed an exact A/B test only with flipping between direct ClusterClient instance from local Silo grainFactory and a regular ClusterClient from ClientBuilder. Only the direct ClusterClient was causing unmanaged memory growth.

**Code samples:**
- silo hostBuilder code
```
IHostBuilder siloHostBuilder = Host.CreateDefaultBuilder()
.UseOrleans((context, builder) =>
{
builder
.Configure(options =>
{
options.SiloName = serviceName;
})
.Configure(options =>
{
options.FallbackSerializationProvider = typeof(Orleans.Serialization.ILBasedSerializer).GetTypeInfo();
options.SerializationProviders.Add(typeof(Orleans.Serialization.ProtobufSerializer).GetTypeInfo());
})
.Configure(options =>
{
options.ClusterId = serviceName.ToMD5HexString();
options.ServiceId = serviceName.ToMD5HexString();
})
.Configure(options =>
{
})
.Configure(options =>
{
options.CollectionAge = actorMaxIdleTime;
})
.ConfigureApplicationParts(parts =>
{
parts = parts.AddApplicationPart(typeof(IInternalActorModule).Assembly);
parts = parts.AddApplicationPart(service.ServiceAssembly).WithReferences();
})
.Configure(options =>
{
options.AddConsumer();
})
.Configure(options =>
{
options.OnLocalSend = OrleansTelemetryConsumer.OnLocalSend;
options.OnRpcSend = OrleansTelemetryConsumer.OnRpcSend;
})
.ConfigureEndpoints(servicePorts.Port_OrleansSilo, servicePorts.Port_OrleansGateway)
.AddPlacementDirector()
.AddPlacementDirector()
.AddOutgoingGrainCallFilter(new ClientGrainCallFilter())

.ConfigureLogging(logging =>
{
var provider = new LoggerProvider("Orleans-Silo", () => Logging.SiloLogLevel);
logging.AddProvider(GetOrAddExtensionLoggerProvider(provider.ProviderName, provider));
});

builder = builder.UseConsulClustering(options =>
{
options.Address = new Uri(consulConnectionString);
options.KvRootFolder = $"Orleans/{serviceName}";
});

});

siloHostBuilder.Build();
```
- code to get direct ClusterClient instance from the local silo
```
Silo.Services.GetRequiredService()
```

**Memory profiling results:**
- A/B test
- hourly dotMemory memory snapshot

- correlation between handle count increase and unmanaged heap memory growth
sysinternals handle tool shows a LOT of 'Event'
And this explosive handle count growth **does not** happen when the regular ClusterClient from ClientBuilder.
```
2260: Event
2264: Semaphore
2268: Event
226C: Semaphore
2270: Event
2274: Event
2278: Event
227C: Semaphore
2280: Event
2284: Semaphore
2288: Event
228C: Semaphore
2290: Event
2294: Semaphore
2298: Event
229C: Semaphore
22A0: Event
22A4: Semaphore
22A8: Event
22AC: Semaphore
22B0: Event
22B4: Semaphore
22B8: Event
22BC: Semaphore
22C0: Event
...
```

**Debugging efforts done:**
- dotMemory memory snapshot analysis
- dotnet-dump analyze
- GC settings change: turn on/off ServerGC, tweaked based on https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#systemgcretainvmcomplus_gcretainvm
- Used Handle tool (https://learn.microsoft.com/en-us/sysinternals/downloads/handle) to inspect handles.
a lot of handles were 'Event' types.

We have been trying many memory profiling and debugging, but still couldn't identify the root cause of this non-stop unmanaged memory growth.

- Is there any known issue with the direct ClusterClient for this symptom? Any clue what could it be ?
- any other recommended debugging techniques to find the cause?

Thanks,
Sam