apache / apache/arrow-julia

Allow constructing an Arrow stream/file from columnar data with no column names

Open
#282 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

We have a data source (Relations from our database engine at RelationalAI) that have _columnar data_, but without column names. (We represent a Relation as a Set of Tuples, e.g. `movie_title` relates movie IDs to Titles, so the positions are meaningful but they do not have names.)

We would like to encode this in Arrow as essentially a Vector of columns. In JSON, we would encode this as:
```json
[
[1001, 2232, 3582, 4030],
["The Matrix", "50 First Dates", "I Am Legend", "The Notebook"]
]
```

From what I can tell, this _is_ supported by the Arrow spec, but isn't currently supported by the Arrow.jl package?

This is the understanding my colleague and I have come to of the current situation:

- Looking at the Arrow spec, each RecordBatch message, containing the actual data, is preceded by a Schema message, defining the logical schema of the former. The Schema contains an array of Field types that define the columns of the RecordBatch in proper order. The name property appears to be optional. That would mean we could serialize columns without a name.
- https://github.com/apache/arrow/blob/56d060ca197352f575edced64e6a1fbc9331b336/format/Schema.fbs#L463
- The fields in the Schema message are flattened, see: https://arrow.apache.org/docs/format/Columnar.html#recordbatch-message
- Arrow.jl does support writing unnamed columns, but only if we supply the data row-wise. Then the resulting arrow schema upon loading contains column names like the following: `Symbol("1")` (which is a bit cumbersome to work with in Julia).

Can we work to expose this ability through the Arrow.jl package as well, in the code to construct an Arrow stream from a column-wise data source?

Thanks!

CC: @bachdavi

Contributor guide

No contributing guide indexed for this repository

Research direction

Read the referenced Arrow Schema.fbs definition and the Arrow.jl code that constructs streams from column-wise data. Compare its handling of row-wise unnamed columns with the requested vector-of-columns input and verify how schemas are reconstructed on load. Done means unnamed column vectors can be serialized and loaded while preserving positional columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.