apache / apache/arrow-julia

Lots of allocations when writing DateTimes

Open
#197 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

I was having some crazy GC pauses, and tracked it down to this:
```
using Dates
using Arrow

julia> tstamps = DateTime.(zeros(1_000_000));
julia> @time Arrow.write("test.arrow", (;t=tstamps))
0.055171 seconds (1.00 M allocations: 24.274 MiB)
"test.arrow"
```
There seems to be one allocation per `DateTime`. I'm on julia 1.6.1, Arrow 1.4.1.

Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the Julia 1.6.1 example using DateTime.(zeros(1_000_000)) and Arrow.write("test.arrow", (;t=tstamps)); then trace the DateTime serialization path used by Arrow.write. Done means writing the DateTime column without one allocation per value and confirming the allocation count with @time.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.