`GridPatch` with `pin_memory=True` significant slow-down in following epochs

Reported by a user (@kenza-bouzid) here:

When using num_workers>1 and pin_memory=True, training time increases exponentially over epochs


I had profiled/timed all intermediate steps, and found out that GridPatch was the guilty one

So I timed all intermediate step in the tranform

It turned out that the operation that was taking too long was a memory allocation by np.array
https://github.com/Project-MONAI/MONAI/blob/a2ec3752f54bfc3b40e7952234fbeb5452ed63e3/monai/transforms/spatial/array.py#L3279
I eventually fixed it by setting pin_memory=False,
which I explain by cuda memory allocation being more expensive as it has to be allocated from the pinned memory

Note that I am dealing with particularly large slides ~50kx50k

Any thoughts on this?

Start with monai/transforms/spatial/array.py at the referenced GridPatch code near line 3279. Reproduce the slowdown using num_workers>1, pin_memory=True, and large slides, then profile the transform and its np.array allocation. Done means the reported cross-epoch slowdown is understood and a validated fix or clear limitation is documented.