apache / apache/arrow-swift

Proposal to re-architect Arrow Swift

Open
#154 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
32
Forks
18
Avg merge
19h 14m
Merged PRs (30d)
2

Description

### Describe the enhancement requested

Arrow Swift can be made significantly more usable and performance can be significantly improved through introducing some major changes.

## The problems

### Buffers can only be a single type

Every ArrowArray holds an ArrowData instance, which holds an array of ArrowBuffers. An ArrowBuffer allocates its own memory. This means that IPC cannot be zero-copy, because the buffer cannot be any other type, such as a view over another binary object like Data.

This is compounded by the fact that the ArrowBuffer exposes its memory via:


```swift
public let rawPointer: UnsafeMutableRawPointer


```
Pointer arithmetic is then being done within the subscript of arrays. This is actually duplicated in each array. 
Second, having a public raw pointer like that is dangerous, e.g. it could easily cause use-after-free errors.


Allowing multiple buffer types, e.g. variable length type buffers and fixed type buffers can improve performance and memory safety.

### ArrowTypes

Currently two reference types are used to describe an ArrowType. There is no reason I can think of for An arrow type to be mutable. Arrow types can be represented as a single value type, using an enum. This is likely improve performance and make the library far more usable, with enums offering much better ergonomics and safety, with exhaustiveness checking and simpler equality checks, for example.

## Feasibility

This is definitely feasible because I have already done most of this in Swift Arrow:
https://github.com/willtemperley/swift-arrow/

This version includes support for all types that Arrow Swift covers, plus schema metadata, map types, binary views, binary with 64 bit offsets and nested types with 64 bit offsets.

It also passes Arrow Gold integration tests for these types.

I have compeltely rewritten the buffer infrastructure. Zero-copy IPC is available using views over memory-mapped Data objects.

I have replaced ArrowType with an enum which was adapted from from arrow-rs.

## The challenge

Changing ArrowType to an enum would completely change the public API.
Replacing the buffer infrastructure means an almost complete rewrite of arrays and array builder.

## Caveats

My version does not include C import / export and Arrow Flight support is pending.

## Suggested way forward

I think this would simply need to be a clean-break fork, with a major version bump and a migration guide.
I'm happy to donate any code I've written in Swift Arrow back to Arrow Swift.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the linked swift-arrow implementation with Arrow Swift's current ArrowBuffer, ArrowData, array-builder, and ArrowType APIs. Review the existing discussion and identify the scope of the clean-break fork, including C import/export, Arrow Flight, the major-version change, and migration guide; done requires an agreed re-architecture plan and contribution path.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
data
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.