Deeply nested structs cause long compilation times
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
Arrow tables with deeply nested lists and structs can make Arrow.jl spend too much time in compilation.
I uploaded an *empty* Arrow table that can trigger this behavior: [observation.empty.arrow.gz](https://github.com/apache/arrow-julia/files/14629731/observation.empty.arrow.gz)
A simple program that reads this table and writes it again can take several minutes:
```
julia -e 'using Arrow; t = Arrow.Table("observation.empty.arrow", convert = false); @time Arrow.write("copy-of-observation.empty.arrow", t)'
250.863146 seconds (58.05 M allocations: 3.791 GiB, 1.07% gc time, 99.99% compilation time)
```
Using Julia 1.10.2 and Arrow 2.7.1.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by downloading observation.empty.arrow.gz and reproducing the Julia 1.10.2 command from the report with Arrow 2.7.1. Profile the read-and-write path to identify where compilation time grows for the nested table. Done means the same reproduction no longer takes several minutes to compile, with relevant tests or benchmarks confirming the improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100