JuliaDataCubes / JuliaDataCubes/YAXArrays.jl
extrema(skipmissing(cube)) much slower than extrema(skipmissing(readcubedata(cube)))
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 132
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Description
When I am trying to get the range of the non-missing values it is much faster to read the whole dataset into memory. For only extrema without skipmissing it doesn't make a difference. These timings are on a 15000x15000 array that is on my local computer.
julia> @time extrema(skipmissing(readcubedata(clocal76)))
0.878060 seconds (148 allocations: 891.717 MiB, 3.05% gc time)
(0x00, 0xfe)
julia> @time extrema(skipmissing(clocal76))
56.791334 seconds (900.00 M allocations: 72.180 GiB, 18.44% gc time)
(0x00, 0xfe)
julia> @time extrema(clocal76)
0.827030 seconds (98 allocations: 891.714 MiB, 1.68% gc time)
(missing, missing)
julia> @time extrema(readcubedata(clocal76))
0.842818 seconds (141 allocations: 891.717 MiB, 5.14% gc time)
(missing, missing)
extrema is only the motivating example, but with a simple collect(skipmissing(cube)) I get very similar timings. I suspect that skipmissing somehow indexes into every single index.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the Julia timings for extrema(skipmissing(cube)), extrema(skipmissing(readcubedata(cube))), and collect(skipmissing(cube)) using the 15000x15000 example described in the issue. Start by comparing the skipmissing paths in YAXArrays, then identify why the lazy cube path allocates so much more. Done means skipmissing reductions no longer incur the reported severe slowdown while preserving the shown results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100