Implement dask-specific methods on DataTree
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What is your issue?
xr.Dataset implements a bunch of dask-specific methods, such as __dask_tokenize__ and __dask_graph__. It also obviously has public methods that involve dask such as .compute() and .load().
In DataTree on the other hand, I haven't yet implemented any methods like these, or even written any tests that involve dask! You can probably still use dask with datatree right now, but from dask's perspective the datatree is presumably merely a set of unconnected Dataset objects.
We could choose to implement methods like .load() as just a mapping over the tree, i.e.
def load(self):
for node in self.subtree:
if node.has_data:
node.ds.load()
Most of that should just already work (or work very easily) using map_over_subtree.
There are also special double-underscore methods defined on Dataset
https://docs.dask.org/en/stable/custom-collections.html
Xarray objects satisfy this Collections protocol, so you can do dask.tokenize(xarray_thing), dask.compute(xarray_thing) etc (visualize, persist).
We could add these, but it would be rather nice if someone who understands the double-underscore dask methods really well just took this on. @darothen helpfully started this in https://github.com/xarray-contrib/datatree/pull/196 but it stalled.
@jrbourbeau are you/Coiled interested in submitting a PR to get xarray.DataTree fully integrated with dask?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with DataTree's existing map_over_subtree behavior and the stalled pull request #196, then review dask's custom-collections protocol and the corresponding Dataset methods. Define and test the DataTree behavior for methods such as load, compute, tokenize, graph, visualize, and persist; done means the relevant dask operations work across the tree.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100