Add custom_metadata support for RecordBatch IPC messages
- Dominant language
- Swift
- Stars
- 32
- Forks
- 18
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 2
Description
## Describe the enhancement requested
The Arrow IPC spec supports `custom_metadata` (key-value pairs) on RecordBatch messages via the `Message.custom_metadata` field. The Swift FlatBuffers generated code (`org_apache_arrow_flatbuf_Message`) already has full read/write support for this field, but the Swift `RecordBatch` model, reader, and writer all ignore it.
### Proposed changes
- Add a `customMetadata: [String: String]` property to `RecordBatch` (defaulting to `[:]` for backward compatibility)
- Add `addMetadata` builder methods on `RecordBatch.Builder`
- Serialize metadata into the FlatBuffers `Message` wrapper when writing (covers file, streaming, and Flight paths via `toMessage`)
- Deserialize metadata from the FlatBuffers `Message` wrapper when reading (covers `readFile`, `readStreaming`, and `fromMessage`)
### Design notes
- `[String: String]` matches the pragmatic API used by pyarrow and most Arrow consumers. Duplicate keys from other implementations are deduplicated (last wins). Dictionary order is not preserved across round-trips.
- Flight support comes for free since `toMessage(batch:)` delegates to `writeRecordBatch`.
- `ArrowTable.from(recordBatches:)` does not propagate per-batch metadata, which is expected since `ArrowTable` is a table-level abstraction.
### Files affected
| File | Change |
|------|--------|
| `Sources/Arrow/ArrowTable.swift` | Add `customMetadata` property and builder methods |
| `Sources/Arrow/ArrowWriter.swift` | Serialize metadata into FlatBuffers Message |
| `Sources/Arrow/ArrowReader.swift` | Deserialize metadata from FlatBuffers Message |
| `Tests/ArrowTests/IPCTests.swift` | Round-trip, builder API, cross-language, and multi-batch tests |
I have an implementation ready and will open a PR referencing this issue.
Contributor guide
Research direction
Start with Sources/Arrow/ArrowTable.swift, ArrowWriter.swift, and ArrowReader.swift to trace the RecordBatch builder and the toMessage/fromMessage paths. Run Tests/ArrowTests/IPCTests.swift first, then add coverage for builder use, round-trips, cross-language behavior, and multiple batches. Done means metadata is preserved through the specified file, streaming, and Flight paths without changing table-level propagation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100