JuliaArrays / JuliaArrays/TiledIteration.jl
Implementing IndexStyle IndexTiled
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 85
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
I'm working on HDF5Arrays.jl, for arrays that have their data stored on disk in HDF5 format. I'm thinking of working on something like a TiledIndex or a ChunkedIndex (with corresponding TiledIndices and IndexTiled <: IndexStyle) With those, efficient functions for everything from eachindex to broadcasting to reduce could be written, that iterate tile by tile.
The start of a TiledIndex implementation might look something like:
struct TiledIndex{N}
tile::NTuple{N, Int} # which tile
size::NTuple{N, Int} # size of tile
index::NTuple{N, Int} # index within tile
end
Tuple(t::TiledIndex) = t.index .+ (t.tile .- 1) .* t.size
CartesianIndex(t::TiledIndex) = CartesianIndex(Tuple(t))
struct TiledIndices{N}
tiles::NTuple{N, UnitRange{Int}} # cartesian indices of tiles
size::NTuple{N, Int} # size of each tile
indices::NTuple{N, CustomRange} # Represents indices within tiles
end
# iteration of custom range with start = 2, stop = 1, repeat = 2, length = 3 yields the following
# 2, 3, 1, 2, 3, 1, 2, 3, 1
DiskArrays.jl has already done some of this work on the iteration front, though it's missing the concept of a chunked or tiled index. Also, I'm also not sure if forcing users to subtype something like an AbstractTiledArray is the best path forward. Hence, I'm thinking about using the IndexStyle trait.
One issue is that functions like _mapreducedim! don't currently dispatch on the IndexStyle trait. This is an issue I'm not sure how to deal with.
I'd be happy to help work on this, but I'm not sure how best to proceed.
- Should iterating over arrays tile by tile be implemented via something like an
AbstractTiledArray, or via the existingIndexStyletrait? - If using traits does indeed make more sense, how do we make efficient reducing functions, given that the base functions don't dispatch on IndexStyle?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing iteration work in DiskArrays.jl and Julia's IndexStyle-based iteration, then inspect how _mapreducedim! currently dispatches. Resolve whether tiled iteration should use an AbstractTiledArray or IndexStyle, and define the scope for TiledIndex, TiledIndices, and efficient tile-wise reductions before implementing it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100