SharedMemoryArray is too slow

I've extensively tested the MP implementation in grain and it's very slow.


The $batch in the middle (violet) is the stacking function that takes 21ms to stack batch of 128. So I expect it should be like 25-40ms as a whole but the 'next' call takes whooping 224 ms. On this profile trace above you can see the awaiting, and then deleting the SharedMemoryArrays, dominates the 'next' call.

I can see how awaiting can struggle because workers are not fast enough (and this is not the case) but I can't see why simple deleting operation could take 1+ms for each array. I believe it just kills the purpose of the shared memory array and naive push through 'multiprocessing.Queue' would be much faster.

Any plans on improving the MP implementation?