JuliaData / JuliaData/SentinelArrays.jl

Performance of SentinelArrays

Open
#82 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
22
Forks
19
PR merge metrics
No merged PRs in 30d

Description

In some practical cases SentinelVector is much slower than Vector. For example for data tested in https://bkamins.github.io/julialang/2022/12/23/duckdb.html.

We have:

julia> summary(posts)
"42710197×3 DataFrame"

julia> typeof.(eachcol(posts))
3-element Vector{DataType}:
 SentinelArrays.ChainedVector{Union{Missing, Int64}, SentinelArrays.SentinelVector{Int64, Int64, Missing, Vector{Int64}}}
 SentinelArrays.ChainedVector{Union{Missing, Int64}, SentinelArrays.SentinelVector{Int64, Int64, Missing, Vector{Int64}}}
 SentinelArrays.ChainedVector{Union{Missing, Int64}, SentinelArrays.SentinelVector{Int64, Int64, Missing, Vector{Int64}}}

julia> @time dropmissing(posts);
  0.819397 seconds (137 allocations: 1.822 GiB)

julia> @time dropmissing(copy(posts));
  0.560146 seconds (130 allocations: 2.657 GiB)

and - as you can see - it is faster to copy a data frame (to change sentinel vectors to just Vector) and then do dropmissing than just do dropmissing directly.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported timings for dropmissing on the SentinelVector-backed DataFrame and on copy(posts), using the DuckDB example linked in the issue. Trace the SentinelVector and ChainedVector paths involved in dropmissing, then compare the results against the reported allocations and runtime. Done means the SentinelVector path no longer has the reported performance disadvantage, with evidence from the same comparison.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.