Aligned allocation for faster SIMD

Problem

Intel's optimization manual places strong emphasis on the importance of alignment of memory addresses SIMD registers are stored/loaded from. Significant performance penalties occur when loads/stores are crossing (64 byte) cache line boundaries.

Managed (array pool) arrays give us no guarantees on the alignment of the &arrray[0] base address, which means that all ImageSharp SIMD code suffers from these penalties globally.

Idea

dotnet/runtime#27146 initially proposed a mechanism to control the alignment of objects on the pinned heap, but that feature did not get into the final implementation.

Considering that the earliest release where runtime support might be available is .NET 6.0, and it won't be available in erlier releases anyways, I'm proposing a library-specific solution:

1. Introduce AllocationOptions.AlignedPinned
2. When the flag is set, return pinned or umanaged buffers with a "hacked" base address (idea taken from https://github.com/dotnet/runtime/issues/33244#issuecomment-595848832):

IntPtr baseAddress = GetBaseAddressOfPinnedArrayOrAllocateUnmanagedBuffer();

// Align the pointer
IntPtr alignedAddress = (IntPtr)((nint)(baseAddress + sizeof(IntPtr) + (alignment - 1)) & ~(alignment - 1));

// Store the pointer to the memory baseAddress to free right before the aligned pointer 
*(((IntPtr*)alignedAddress) - 1) = baseAddress;

return AlignedBuffer(alignedAddress);

The question I can't figure out

I want to make sure there are no unwanted side-effects for the GC. What buffers should we use with this trick, considering that the (mid to large) buffers we want to align will be always temporary? Note that max lifetime of these buffers is the run of an image codec or a processing operation, pinned allocations will be typically bound to these kind of big, expensive operations and won't happen within individual primitive steps.

• Should we just pin the arrays of our pool?
• Or use Marshal.Alloc?
• Should we prefer to use the Pinned Heap when it's available?

@tannergooding @saucecontrol any thoughts/concerns?

Trying to also summon @benaadams if it's not too impolite.

Start with the proposed AllocationOptions.AlignedPinned design and the array-pool, pinned-array, and Marshal.Alloc alternatives described here. Review the linked runtime issues for constraints and determine whether a safe allocation strategy and ownership/freeing behavior can be specified; done means the alignment API and GC/lifetime trade-offs have a decided implementation scope.