apache / apache/arrow-dotnet

Support large arrays

Open
#187 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
39
Forks
30
Avg merge
1d 9h
Merged PRs (30d)
16

Description

### Describe the enhancement requested

It's hard for the C# Arrow library to support large arrays. Commonly-used types like `Array`, `Span` and `Memory` use a 32-bit signed integer for the length, meaning that they can't have more than ~2 million elements in them. The `ArrowBuffer` structure that is the foundation of the core `ArrayData` class exposes these types directly. This may make it impossible to support large arrays in a strictly backwards-compatible fashion.

Some previously-opened items related to this feature are:

[https://github.com/apache/arrow/issues/23776](https://github.com/apache/arrow/issues/23776)
[https://github.com/apache/arrow/issues/38086](https://github.com/apache/arrow/issues/38086)
[https://github.com/apache/arrow/pull/38619](https://github.com/apache/arrow/pull/38619)
[https://github.com/apache/arrow/issues/45030](https://github.com/apache/arrow/issues/45030)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.