JuliaArrays / JuliaArrays/AxisArrays.jl
Faster indexing with Categorical axes
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 207
- Forks
- 41
- PR merge metrics
- No merged PRs in 30d
Description
With named columns, indexing based on a symbol slows down tight loops. DataFrames has the same problem. It's tough to get fast, type-stable code for this. Here is an example:
using AxisArrays
N = 10_000_000
A = AxisArray(rand(10000000,2), (1:N,[:a,:b]));
function f(x, j)
res = 0.0
for i in 1:size(x,1)
res += x[i,j]
end
res
end
@time f(A, 2) # elapsed time: 0.028395385 seconds (117 kB allocated)
@time f(A, :b) # elapsed time: 0.147722792 seconds (364 kB allocated)
I tried using types as column names to speed things up, but the resulting code isn't type stable, and it's even slower. Here's my attempt:
## Bunch of kludgy stuff
AxisArrays.checkaxis{T}(::Type{Val{T}}) = true
stagedfunction AxisArrays.axisindexes{T,S}(::Type{Val{T}}, ::Type{Val{S}})
i = findfirst(T, S)
:($i)
end
Base.length{T}(::Type{Val{T}}) = length(T)
Base.size{T}(::Type{Val{T}}) = size(T)
Base.size{T}(::Type{Val{T}},i) = size(T,i)
B = AxisArray{Float64,2,Array{Float64,2},(:row,:col),(UnitRange{Int64},DataType)}(rand(N,2), (1:N,Val{(:a,:b)}));
@time f(B, Val{:b}) # elapsed time: 0.982083383 seconds (461 MB allocated, 3.26% gc time in 20 pauses with 0 full sweep)
@time f(B, 2) # elapsed time: 0.019993164 seconds (88 kB allocated)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue's f(A, 2) and f(A, :b) benchmarks with the supplied AxisArrays example, then inspect the indexing path for named axes. Done means symbol-based indexing is faster and type-stable without regressing integer indexing; the issue names no files or tests to run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100