Kotlin / Kotlin/dataframe

Preserve nested structure when writing `FrameColumn`s and `List`-typed columns to Arrow

Open
#1,913 0 comments 0 reactions 1 assignee View on GitHub

@koperagen is already working on this.

Since Jul 1, 2026.

enhancement
Dominant language
Kotlin
Stars
1.1k
Forks
83
Avg merge
4d 12h
Merged PRs (30d)
30

Description

Follow up of #271.

Problem

Reading of ColumnGroups and FrameColumns from Arrow works already. Writing ColumnGroups to Arrow works as well.

However, writing of FrameColumns to Arrow is still implemented via toString, which leads to an incorrect Arrow schema (due to lost structure of nested types) and to losing data (toString trims it). The same problem appears with List-typed columns: writing to Arrow transforms List to String.

For example, given this dataframe:

val dataFrame = DataFrame.read("https://raw.githubusercontent.com/phodal-archive/apache-arrow-chapi-demo/master/data/0_codes.json")

its schema will look in the following way:

dataFrame.schema().print()
NodeName: String
Module: String
Type: String
Package: String
FilePath: String
Fields: *
    TypeType: String
    TypeKey: String
    Modifiers: List<String>
Implements: List<String>
Functions: *
    Name: String
    Package: String
    ReturnType: String
    Parameters: *
        TypeValue: String
        TypeType: String
    FunctionCalls: *
        Package: String?
        NodeName: String
        FunctionName: String
        Position:
            StartLine: Int
            StartLinePosition: Int
            StopLine: Int
            StopLinePosition: Int
        Parameters: *
            TypeValue: String
            TypeType: String

    Position:
        StartLine: Int
        StartLinePosition: Int
        StopLine: Int
        StopLinePosition: Int
    LocalVariables: *
        TypeValue: String
        TypeType: String
    IsConstructor: Boolean?
Imports: *
    Source: String
    AsName: String
Position:
    StartLine: Int
    StopLine: Int

However, when we obtain an Arrow schema:

val toArrowSchema = dataFrame.columns().toArrowSchema()
println(toArrowSchema.toJson())

We get the following:

{
  "fields" : [ {
    "name" : "NodeName",
    "nullable" : false,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "Module",
    "nullable" : false,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "Type",
    "nullable" : false,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "Package",
    "nullable" : false,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "FilePath",
    "nullable" : false,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "Fields",
    "nullable" : true,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "Implements",
    "nullable" : true,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "Functions",
    "nullable" : true,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "Imports",
    "nullable" : true,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "Position",
    "nullable" : false,
    "type" : {
      "name" : "struct"
    },
    "children" : [ {
      "name" : "StartLine",
      "nullable" : false,
      "type" : {
        "name" : "int",
        "bitWidth" : 32,
        "isSigned" : true
      },
      "children" : [ ]
    }, {
      "name" : "StopLine",
      "nullable" : false,
      "type" : {
        "name" : "int",
        "bitWidth" : 32,
        "isSigned" : true
      },
      "children" : [ ]
    } ]
  } ]
}

If we write the original dataframe to Arrow, then read it back and print its schema:

dataFrame.writeArrowFeather(File("codes.arrow"))
val readFromArrow = DataFrame.readArrowFeather("codes.arrow")
readFromArrow.schema().print()

We obtain the following schema:

NodeName: String
Module: String
Type: String
Package: String
FilePath: String
Fields: String
Implements: String
Functions: String
Imports: String
Position:
    StartLine: Int
    StopLine: Int

The children in nested types represented by FrameColumns are lost because the nested frames are represented as regular strings.
Additionally, the "Implements" column, which originally had the type List<String>, now has the type String.

What needs to be done

We need to implement support of writing FrameColumns and List-typed columns to Arrow which preserves the structure and the data of the column.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.