apache / apache/arrow-julia

Subsequent reads are not faster

Open
#268 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

Perhaps I'm misunderstanding `mmap`, but I would expect that `copy(df)` would be much faster the second time around, because `mmap` has by then lazily loaded the data into memory. On our 0.5GB dataset, we get

```julia
julia> df = DataFrame(Arrow.Table("some_file.arrow")); 0
0

julia> @time copy(df); 0
1.412632 seconds (1.03 k allocations: 616.774 MiB, 1.76% gc time)
0

julia> @time copy(df); 0
1.451533 seconds (1.03 k allocations: 616.774 MiB, 7.55% gc time)
0

julia> @time copy(df); 0
1.194862 seconds (1.03 k allocations: 616.774 MiB, 0.29% gc time)
0
```

In contrast, with plain `mmap` I see a factor of nearly 3.

Details

```julia
julia> m, n = 5_000_000, 300
(5000000, 300)

julia> A = rand(1:20, m, n);

julia> s = open("mmap.bin", "w+")
IOStream()

julia> write(s, A)
12000000000

julia> close(s)

julia> A = nothing
```

then we read it with mmap

```julia
julia> using Mmap

julia> m, n = 5_000_000, 300
(5000000, 300)

julia> s = open("mmap.bin", "r")
IOStream()

julia> A2 = Mmap.mmap(s, Matrix{Int}, (m,n)); 0
0

julia> @time copy(A2); 0
17.176765 seconds (536 allocations: 11.176 GiB, 0.05% gc time, 0.01% compilation time)
0

julia> @time copy(A2); 0
6.409250 seconds (2 allocations: 11.176 GiB, 0.03% gc time)
0
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the timings from DataFrame(Arrow.Table("some_file.arrow")) and compare them with Mmap.mmap and copy(A2). Start at the Arrow.Table loading path and the copy behavior shown in the Julia examples. Done means determining whether repeated reads should improve and clearly identifying the cause or expected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.