Preserve nested structure when writing `FrameColumn`s and `List`-typed columns to Arrow
@koperagen is already working on this.
Since Jul 1, 2026.
- Dominant language
- Kotlin
- Stars
- 1.1k
- Forks
- 83
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 30
Description
Follow up of #271.
Problem
Reading of ColumnGroups and FrameColumns from Arrow works already. Writing ColumnGroups to Arrow works as well.
However, writing of FrameColumns to Arrow is still implemented via toString, which leads to an incorrect Arrow schema (due to lost structure of nested types) and to losing data (toString trims it). The same problem appears with List-typed columns: writing to Arrow transforms List to String.
For example, given this dataframe:
val dataFrame = DataFrame.read("https://raw.githubusercontent.com/phodal-archive/apache-arrow-chapi-demo/master/data/0_codes.json")
its schema will look in the following way:
dataFrame.schema().print()
NodeName: String
Module: String
Type: String
Package: String
FilePath: String
Fields: *
TypeType: String
TypeKey: String
Modifiers: List<String>
Implements: List<String>
Functions: *
Name: String
Package: String
ReturnType: String
Parameters: *
TypeValue: String
TypeType: String
FunctionCalls: *
Package: String?
NodeName: String
FunctionName: String
Position:
StartLine: Int
StartLinePosition: Int
StopLine: Int
StopLinePosition: Int
Parameters: *
TypeValue: String
TypeType: String
Position:
StartLine: Int
StartLinePosition: Int
StopLine: Int
StopLinePosition: Int
LocalVariables: *
TypeValue: String
TypeType: String
IsConstructor: Boolean?
Imports: *
Source: String
AsName: String
Position:
StartLine: Int
StopLine: Int
However, when we obtain an Arrow schema:
val toArrowSchema = dataFrame.columns().toArrowSchema()
println(toArrowSchema.toJson())
We get the following:
{
"fields" : [ {
"name" : "NodeName",
"nullable" : false,
"type" : {
"name" : "utf8"
},
"children" : [ ]
}, {
"name" : "Module",
"nullable" : false,
"type" : {
"name" : "utf8"
},
"children" : [ ]
}, {
"name" : "Type",
"nullable" : false,
"type" : {
"name" : "utf8"
},
"children" : [ ]
}, {
"name" : "Package",
"nullable" : false,
"type" : {
"name" : "utf8"
},
"children" : [ ]
}, {
"name" : "FilePath",
"nullable" : false,
"type" : {
"name" : "utf8"
},
"children" : [ ]
}, {
"name" : "Fields",
"nullable" : true,
"type" : {
"name" : "utf8"
},
"children" : [ ]
}, {
"name" : "Implements",
"nullable" : true,
"type" : {
"name" : "utf8"
},
"children" : [ ]
}, {
"name" : "Functions",
"nullable" : true,
"type" : {
"name" : "utf8"
},
"children" : [ ]
}, {
"name" : "Imports",
"nullable" : true,
"type" : {
"name" : "utf8"
},
"children" : [ ]
}, {
"name" : "Position",
"nullable" : false,
"type" : {
"name" : "struct"
},
"children" : [ {
"name" : "StartLine",
"nullable" : false,
"type" : {
"name" : "int",
"bitWidth" : 32,
"isSigned" : true
},
"children" : [ ]
}, {
"name" : "StopLine",
"nullable" : false,
"type" : {
"name" : "int",
"bitWidth" : 32,
"isSigned" : true
},
"children" : [ ]
} ]
} ]
}
If we write the original dataframe to Arrow, then read it back and print its schema:
dataFrame.writeArrowFeather(File("codes.arrow"))
val readFromArrow = DataFrame.readArrowFeather("codes.arrow")
readFromArrow.schema().print()
We obtain the following schema:
NodeName: String
Module: String
Type: String
Package: String
FilePath: String
Fields: String
Implements: String
Functions: String
Imports: String
Position:
StartLine: Int
StopLine: Int
The children in nested types represented by FrameColumns are lost because the nested frames are represented as regular strings.
Additionally, the "Implements" column, which originally had the type List<String>, now has the type String.
What needs to be done
We need to implement support of writing FrameColumns and List-typed columns to Arrow which preserves the structure and the data of the column.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.