apache / apache/arrow-julia

Deeply nested structs cause long compilation times

Open
#503 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

Arrow tables with deeply nested lists and structs can make Arrow.jl spend too much time in compilation.

I uploaded an *empty* Arrow table that can trigger this behavior: [observation.empty.arrow.gz](https://github.com/apache/arrow-julia/files/14629731/observation.empty.arrow.gz)

A simple program that reads this table and writes it again can take several minutes:
```
julia -e 'using Arrow; t = Arrow.Table("observation.empty.arrow", convert = false); @time Arrow.write("copy-of-observation.empty.arrow", t)'
250.863146 seconds (58.05 M allocations: 3.791 GiB, 1.07% gc time, 99.99% compilation time)
```

Using Julia 1.10.2 and Arrow 2.7.1.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by downloading observation.empty.arrow.gz and reproducing the Julia 1.10.2 command from the report with Arrow 2.7.1. Profile the read-and-write path to identify where compilation time grows for the nested table. Done means the same reproduction no longer takes several minutes to compile, with relevant tests or benchmarks confirming the improvement.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.