apache / apache/arrow-swift

Add custom_metadata support for RecordBatch IPC messages

Open
#135 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
32
Forks
18
Avg merge
19h 14m
Merged PRs (30d)
2

Description

## Describe the enhancement requested

The Arrow IPC spec supports `custom_metadata` (key-value pairs) on RecordBatch messages via the `Message.custom_metadata` field. The Swift FlatBuffers generated code (`org_apache_arrow_flatbuf_Message`) already has full read/write support for this field, but the Swift `RecordBatch` model, reader, and writer all ignore it.

### Proposed changes

- Add a `customMetadata: [String: String]` property to `RecordBatch` (defaulting to `[:]` for backward compatibility)
- Add `addMetadata` builder methods on `RecordBatch.Builder`
- Serialize metadata into the FlatBuffers `Message` wrapper when writing (covers file, streaming, and Flight paths via `toMessage`)
- Deserialize metadata from the FlatBuffers `Message` wrapper when reading (covers `readFile`, `readStreaming`, and `fromMessage`)

### Design notes

- `[String: String]` matches the pragmatic API used by pyarrow and most Arrow consumers. Duplicate keys from other implementations are deduplicated (last wins). Dictionary order is not preserved across round-trips.
- Flight support comes for free since `toMessage(batch:)` delegates to `writeRecordBatch`.
- `ArrowTable.from(recordBatches:)` does not propagate per-batch metadata, which is expected since `ArrowTable` is a table-level abstraction.

### Files affected

| File | Change |
|------|--------|
| `Sources/Arrow/ArrowTable.swift` | Add `customMetadata` property and builder methods |
| `Sources/Arrow/ArrowWriter.swift` | Serialize metadata into FlatBuffers Message |
| `Sources/Arrow/ArrowReader.swift` | Deserialize metadata from FlatBuffers Message |
| `Tests/ArrowTests/IPCTests.swift` | Round-trip, builder API, cross-language, and multi-batch tests |

I have an implementation ready and will open a PR referencing this issue.

Contributor guide

Open the contributing guide

Research direction

Start with Sources/Arrow/ArrowTable.swift, ArrowWriter.swift, and ArrowReader.swift to trace the RecordBatch builder and the toMessage/fromMessage paths. Run Tests/ArrowTests/IPCTests.swift first, then add coverage for builder use, round-trips, cross-language behavior, and multiple batches. Done means metadata is preserved through the specified file, streaming, and Flight paths without changing table-level propagation.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.