apache / apache/arrow-rs

AppendableRecordBatch

Open
#3,142 4 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

At Space and Time, we would like to make an OLTP columnar DB based on Arrow (see noisepage for precedent). Unfortunately, Arrow RecordBatches are immutable by design.

**Describe the solution you'd like**

Fortunately however, append-only and immutable are not necessarily at odds, as long as one takes immutable RecordBatch snapshots at a point-in-time of the append-only version. See [this gist](https://gist.github.com/avantgardnerio/60a53a3481f3d7844efa12346ef6f814) for a working PoC implementation:

![Screenshot from 2022-11-19 09-46-31](https://user-images.githubusercontent.com/3855243/202862296-8b90220d-0828-411b-b83c-4215d8633b10.png)

**Describe alternatives you've considered**

1. using arrow2
2. trying some trickery with making the `MutableBuffer` work, vs preallocating ones from the immutable package

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the linked working PoC and the existing Arrow RecordBatch, MutableBuffer, and immutable-package APIs. Establish the required append-only snapshot semantics and tests before implementation; done means the project has an agreed scope for supporting appendable batches without violating immutable snapshots.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.