mcabbott / mcabbott/AxisKeys.jl

Slicing with larger key vectors is slow

Open
#146 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
154
Forks
31
PR merge metrics
No merged PRs in 30d

Description

Thanks for this great package! I find myself regularly slicing large KeyedArray matrices with large vectors of string keys (about 10% of the matrix). This is unfortunately currently slow:

using Random, AxisKeys, BenchmarkTools
A = KeyedArray(zeros(100000, 100), sid=["S$i" for i in 1:100000], oid=["O$i" for i in 1:100])
sub_sids = rand(axiskeys(A, 1), 10000)
A_sub_slow = @btime A[Key(sub_sids), :] 
# run time: 6.156 s

On my real data it can take minutes, which is why I regularly find myself using an indexin workaround:

A_sub_fast = @btime begin
    sub_sids_idx = indexin(sub_sids, axiskeys(A, 1))
    A[sub_sids_idx, :]
end 
# run time: 26.203 ms

@assert A_sub_slow == A_sub_fast

The slow method becomes much faster when slicing the matrix at the beginning (sub_sids = axiskeys(A, 1)[1:10000],
255.530 ms) and much slower when slicing at the end (sub_sids = axiskeys(A, 1)[end-10000:end],
18.904 s). The fast method is faster in all scenarios: 12.335 ms (slice beginning), 15.037 ms (slice end).

Only when performing small slices (100 elements) at the beginning can I see advantages of the default method (36.438 μs vs 77.185 μs). I may be missing something, but could the current method perhaps make use of indexin internally?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the KeyedArray indexing path handling A[Key(sub_sids), :] and inspect how it resolves string keys. Reproduce the issue with the provided benchmark, compare it with the indexin workaround, and consider the work complete when large key-vector slicing matches the workaround's results without the reported slowdown.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.