CommunityToolkit / CommunityToolkit/dotnet
[HighPerformance] CommunityToolkit.HighPerformance.Buffers.LargeArray<T>
- Dominant language
- C#
- Stars
- 3.8k
- Forks
- 400
- PR merge metrics
- No merged PRs in 30d
Description
### Overview
Today, [.NET does not support array dimensions larger than 0x7FFFFFC7](https://github.com/dotnet/runtime/issues/12221). However, it is possible to implement a `LargeArray` type by increasing the size of the array element (for example, an array of `T, T` tuples instead of an array of `T`s) or allocating a multi-dimensional array.
### API breakdown
```cs
using System;
using System.Collections;
using System.Collections.Generic;
using System.ComponentModel;
namespace CommunityToolkit.HighPerformance.Buffers;
public abstract class LargeArray
{
public nuint Length { get; }
// Mimic the APIs on System.GC
public static LargeArray Allocate(nuint length, bool pinned = false);
public static LargeArray AllocateUninitialized(nuint length, bool pinned = false);
// Only LargeArray should inherit from this type
private protected LargeArray();
// Intended to aid serializers and reflection use cases
public abstract Type GetElementType(); // Implemented as 'return typeof(T)' on LargeArray
public abstract object? GetValue(nuint index);
public abstract void SetValue(object? value, nuint index);
// Returns 'ref null' for an empty array
[EditorBrowsable(EditorBrowsableState.Never)]
public ref byte GetPinnableReference();
// LargeArrayMarshal.TryGetArray, InvalidCastException on failure
public static explicit operator Array(LargeArray array);
}
public sealed class LargeArray : LargeArray
{
public LargeArray(nuint length);
public Span AsSpan(nuint start, int length);
public Memory AsMemory(nuint start, int length);
public SegmentEnumerator EnumerateSegments();
public override Type GetElementType();
// Hidden to discourage use over the indexers when you have a concrete LargeArray
[EditorBrowsable(EditorBrowsableState.Never)]
public override object? GetValue(nuint index);
[EditorBrowsable(EditorBrowsableState.Never)]
public override void SetValue(object? value, nuint index);
// Throws 'IndexOutOfRangeException' for bad indices
public ref T this[nint index] { get; }
public ref T this[nuint index] { get; }
// Returns 'ref null' for an empty array
[EditorBrowsable(EditorBrowsableState.Never)]
public new ref T GetPinnableReference();
// LargeArrayMarshal.TryGetArray, InvalidCastException on failure
public static explicit operator T[](LargeArray array);
public static explicit operator Array(LargeArray array);
public struct SegmentEnumerator
{
public LargeArraySegment Current { get; }
public bool MoveNext();
public SegmentEnumerator GetEnumerator();
}
}
// Represents at most int.MaxValue elements after 'Offset'
public readonly struct LargeArraySegment
{
public LargeArray Array { get; }
public nuint Offset { get; }
public LargeArraySegment(LargeArray array, nuint offset);
// Array.AsSpan(Offset, (int)nuint.Min(int.MaxValue, Array.Length - Offset))
public Span Span { get; }
// Included to enable 'foreach ((var offset, var span) in array.EnumerateSegments())'
public void Deconstruct(out nuint offset, out Span span);
// Included to enable 'foreach (Span span in array.EnumerateSegments())'
public static explicit operator Span(LargeArraySegment segment);
public static explicit operator ReadOnlySpan(LargeArraySegment segment);
}
public static class LargeArrayMarshal
{
public static ref byte GetLargeArrayDataReference(LargeArray array);
public static ref T GetLargeArrayDataReference(LargeArray array);
public static LargeArray CreateLargeArray(Array buffer, nuint length);
// Returns the underlying storage array. This is not necessarily a T[].
public static Array GetBuffer(LargeArray array);
// If the LargeArray is backed by a regular T[] that matches in length, this will succeed.
public static bool TryGetArray(LargeArray array, [NotNullWhen(true)] out Array? result);
// If the LargeArray is backed by a regular T[] that matches in length, this will succeed.
public static bool TryGetArray(LargeArray array, [NotNullWhen(true)] out T[]? result);
}
public static class LargeArrayExtensions
{
// Safe version of 'LargeArrayMarshal.CreateLargeArray(array, (uint)array.Length)'
public static LargeArray AsLargeArray(this T[] array);
// Implemented using 'foreach ((var offset, var span) in array.EnumerateSegments())'
public static nint IndexOf(this LargeArray array, T value) where T : IEquatable;
public static nint IndexOf(this LargeArray array, ReadOnlySpan value) where T : IEquatable;
public static nint LastIndexOf(this LargeArray array, T value) where T : IEquatable;
public static nint LastIndexOf(this LargeArray array, ReadOnlySpan value) where T : IEquatable;
// ...TBD
}
```
### Usage example
```cs
var array = new LargeArray(someLargeLength);
array[hugeIndex] = 50;
```
```cs
var array = new LargeArray(someLargeLength);
var memory = array.AsMemory(hugeIndex, readCount);
await reader.ReadAsync(memory, cancellationToken);
```
### Breaking change?
No
### API design questions
* Should `LargeArray` provide `int`, `uint`, etc. indexer properties?
* Should `LargeArray` implement `IEnumerable`?
* Are there any important APIs missing from the proposal that should be included? (e.g., a `CopyTo` equivalent)
* Would it be desirable for `LargeArrayMarshal.CreateLargeArray` to perform validation on the input (element size, etc.)?
* What should be done about `GetHashCode` and equality?
### Additional context
A proof-of-concept implementation of `LargeArray` by myself and @hamarb123 can be found [here](https://gist.github.com/DaZombieKiller/e74217cc7a28f1168db970e7a3c5cb43).
### Help us help you
Yes, but only if others can assist
Contributor guide
Assessment
This issue has not been assessed yet.