JuliaData / JuliaData/SentinelArrays.jl
Performance of SentinelArrays
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 22
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
In some practical cases SentinelVector is much slower than Vector. For example for data tested in https://bkamins.github.io/julialang/2022/12/23/duckdb.html.
We have:
julia> summary(posts)
"42710197×3 DataFrame"
julia> typeof.(eachcol(posts))
3-element Vector{DataType}:
SentinelArrays.ChainedVector{Union{Missing, Int64}, SentinelArrays.SentinelVector{Int64, Int64, Missing, Vector{Int64}}}
SentinelArrays.ChainedVector{Union{Missing, Int64}, SentinelArrays.SentinelVector{Int64, Int64, Missing, Vector{Int64}}}
SentinelArrays.ChainedVector{Union{Missing, Int64}, SentinelArrays.SentinelVector{Int64, Int64, Missing, Vector{Int64}}}
julia> @time dropmissing(posts);
0.819397 seconds (137 allocations: 1.822 GiB)
julia> @time dropmissing(copy(posts));
0.560146 seconds (130 allocations: 2.657 GiB)
and - as you can see - it is faster to copy a data frame (to change sentinel vectors to just Vector) and then do dropmissing than just do dropmissing directly.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported timings for dropmissing on the SentinelVector-backed DataFrame and on copy(posts), using the DuckDB example linked in the issue. Trace the SentinelVector and ChainedVector paths involved in dropmissing, then compare the results against the reported allocations and runtime. Done means the SentinelVector path no longer has the reported performance disadvantage, with evidence from the same comparison.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100