mcabbott / mcabbott/AxisKeys.jl
Slicing with larger key vectors is slow
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 154
- Forks
- 31
- PR merge metrics
- No merged PRs in 30d
Description
Thanks for this great package! I find myself regularly slicing large KeyedArray matrices with large vectors of string keys (about 10% of the matrix). This is unfortunately currently slow:
using Random, AxisKeys, BenchmarkTools
A = KeyedArray(zeros(100000, 100), sid=["S$i" for i in 1:100000], oid=["O$i" for i in 1:100])
sub_sids = rand(axiskeys(A, 1), 10000)
A_sub_slow = @btime A[Key(sub_sids), :]
# run time: 6.156 s
On my real data it can take minutes, which is why I regularly find myself using an indexin workaround:
A_sub_fast = @btime begin
sub_sids_idx = indexin(sub_sids, axiskeys(A, 1))
A[sub_sids_idx, :]
end
# run time: 26.203 ms
@assert A_sub_slow == A_sub_fast
The slow method becomes much faster when slicing the matrix at the beginning (sub_sids = axiskeys(A, 1)[1:10000],
255.530 ms) and much slower when slicing at the end (sub_sids = axiskeys(A, 1)[end-10000:end],
18.904 s). The fast method is faster in all scenarios: 12.335 ms (slice beginning), 15.037 ms (slice end).
Only when performing small slices (100 elements) at the beginning can I see advantages of the default method (36.438 μs vs 77.185 μs). I may be missing something, but could the current method perhaps make use of indexin internally?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the KeyedArray indexing path handling A[Key(sub_sids), :] and inspect how it resolves string keys. Reproduce the issue with the provided benchmark, compare it with the indexin workaround, and consider the work complete when large key-vector slicing matches the workaround's results without the reported slowdown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100