NCAS-CMS / NCAS-CMS/PyActiveStorage

Method to query if active storage reductions are available.

Open
#56 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
6
Forks
2
PR merge metrics
No merged PRs in 30d

Description

When incorporating active storage into Dask, the Dask graph needs to be modified non-lazily to account for the fact that some of the work is being done externally to Dask, i.e. on the server where the data is [*].

If cf-python thinks that active storage operation are not possible, then it simply doesn't modify the graph. There are many reasons why active storage is not deemed OK, such as the data has already been operated on (f += 2), the chunks to not point to files on disk, but the relevant one here is that the file reside at a location that doesn't support active reductions.

So, it would be great to have a method of Active that can tell us if a given file can be reduced actively, something like (notional API):

>>> a = Active('/path/to/file.nc')
>>> a.isactive()
True # or False

I imagine that this could entail some try ... except ... approach whereby we assume the file is active, send off some mofided URI that returns True iff it is possible.

[*] (The detail of this is that the chunk reduction function used by dask.array.reduction needs to be changed from the usual function that expects to do some work (e.g. np.max) to the identity function that does no work (e.g. lambda x: x). This has to be done prior to the compute().)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Active class and trace how it determines whether active reductions are possible before Dask computes the graph. Review the interaction with dask.array.reduction and the modified URI approach described in the issue. Done means a method can report whether a given file supports active reduction without changing the graph incorrectly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.