pydata / pydata/xarray

Rename `ChunkManager` to `ComputeManager`

Open
#9,435 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

API design topic-chunked-arrays
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What is your issue?

In https://github.com/pydata/xarray/issues/8733#issuecomment-2251861380 and https://github.com/pydata/xarray/pull/9286#discussion_r1694656560 it's become fairly clear that the ChunkManager abstraction isn't quite right - it's "too greedy" as @dcherian said. #9286 will fix this by removing .chunks and .rechunk from the ChunkManager's responsibilities, but the result will be a "ChunkManager" that doesn't explicitly handle chunks!

I think a better name to describe the new interface is a "ComputeManager", as it still handles the creation of lazily-computed parallel arrays, distribution of computation over parts of those arrays, and triggering the materialization of the arrays.

JAX is also an interesting potential use case because there you don't have chunks, but you do still have to manage dividing computation up over multiple devices. See https://github.com/pydata/xarray/pull/9286#discussion_r1694656560

Renaming ChunkManagerEntrypoint to ComputeManagerEntrypoint will be a breaking change but:
a) this is a very advanced feature,
b) the docs for it have a fat "experimental" warning on them,
c) I'm only aware of 2 libraries using this outside of xarray itself: cubed (tagging @tomwhite), and @hmaarrfk's chunked data structure. The dask ChunkManager ships with xarray, so there is no breaking change there. (Users may have to pip install again to re-register entrypoints if upgrading a development version of xarray inside existing environments though.)

I'm separating this out from #9286 because that PR shouldn't be a breaking change, and the follow-up that closes this issue will be the minimal possible breaking change (i.e. just renaming ChunkManagerEntrypoint -> ComputeManagerEntrypoint).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing PR #9286 and the internals documentation at docs.xarray.dev/en/stable/internals/chunked-arrays.html, then search the repository for ChunkManagerEntrypoint and related references. The follow-up should make the minimal breaking rename to ComputeManagerEntrypoint and update the affected references and entry-point registration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.