Client performance significantly worse than streams performance

Orleans 3.0.2, .NET Core 3.1, Windows 10 b1809
# Scenario
I have a `Core` service that implicitly subscribes to data from a custom Kafka stream provider.

I also need to `Backload` historical events from different sources (files, databases) into the `Core` service.

My `Core` service grains use `IGrainWithGuidKey`, and the `Core` stream provider batch container and `Backload` service both call a simple piece of code that can take a `byte[]` for the event and quickly spit out the correct `streamId` the event needs to be delivered to (there's a ratio of about 1-12 events per grain p.a.).

# Expected

My file backloader has thousands of archives (~20MB each compressed, 15x that decompressed) that contain thousands of events. Loading this data obviously parallelises extremely well and is only really limited by storage I/O. I would expect this to perform as fast as, or faster than, the stream provider. Psuedo code:

````
await client.Connect();
foreach (var file in files.AsParallel().WithDegreeOfParallelism(parallelism))
var archive = open(file)
foreach (var entry in archive)
var streamId = streamAdapter(entry.bytes)
var grain = client.GetGrain(streamId)
await grain.OnXXX(entry.bytes);
````

# Actual

When I run the `Backload` service, despite fine tuning the `parallelism` parameter I can never exceed 2.5% of the raw throughput I see (i.e. without the `await grain.OnXXX(entry.bytes)` line). This translates into only 10% of the streams performance in `Core` itself.

I changed the code to a client per-thread:

````
foreach (var file in files.AsParallel().WithDegreeOfParallelism(parallelism))
await client.Connect();
...
````

And throughput is exactly the same.

Have I missed something obvious?

Reproduce the backload scenario on Orleans 3.0.2 with the custom Kafka stream provider, comparing raw archive throughput with the awaited grain.OnXXX(entry.bytes) calls. Start by isolating client-to-grain invocation behavior and the effect of parallelism; done means identifying why backloading is slower and documenting a verified cause or remedy.

Written by the indexing model from the issue text.