JuliaArrays / JuliaArrays/TiledIteration.jl

Implementing IndexStyle IndexTiled

Open
#24 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
85
Forks
8
PR merge metrics
No merged PRs in 30d

Description

I'm working on HDF5Arrays.jl, for arrays that have their data stored on disk in HDF5 format. I'm thinking of working on something like a TiledIndex or a ChunkedIndex (with corresponding TiledIndices and IndexTiled <: IndexStyle) With those, efficient functions for everything from eachindex to broadcasting to reduce could be written, that iterate tile by tile.

The start of a TiledIndex implementation might look something like:

struct TiledIndex{N}
    tile::NTuple{N, Int} # which tile
    size::NTuple{N, Int} # size of tile
    index::NTuple{N, Int} # index within tile
end

Tuple(t::TiledIndex) = t.index .+ (t.tile .- 1) .* t.size
CartesianIndex(t::TiledIndex) = CartesianIndex(Tuple(t))

struct TiledIndices{N}
    tiles::NTuple{N, UnitRange{Int}} # cartesian indices of tiles
    size::NTuple{N, Int} # size of each tile
    indices::NTuple{N, CustomRange} # Represents indices within tiles
end

# iteration of custom range with start = 2, stop = 1, repeat = 2, length = 3 yields the following
# 2, 3, 1, 2, 3, 1, 2, 3, 1

DiskArrays.jl has already done some of this work on the iteration front, though it's missing the concept of a chunked or tiled index. Also, I'm also not sure if forcing users to subtype something like an AbstractTiledArray is the best path forward. Hence, I'm thinking about using the IndexStyle trait.

One issue is that functions like _mapreducedim! don't currently dispatch on the IndexStyle trait. This is an issue I'm not sure how to deal with.

I'd be happy to help work on this, but I'm not sure how best to proceed.

  • Should iterating over arrays tile by tile be implemented via something like an AbstractTiledArray, or via the existing IndexStyle trait?
  • If using traits does indeed make more sense, how do we make efficient reducing functions, given that the base functions don't dispatch on IndexStyle?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing iteration work in DiskArrays.jl and Julia's IndexStyle-based iteration, then inspect how _mapreducedim! currently dispatches. Resolve whether tiled iteration should use an AbstractTiledArray or IndexStyle, and define the scope for TiledIndex, TiledIndices, and efficient tile-wise reductions before implementing it.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.