dask / dask/distributed

Documentation / API to check if a Dask collection is persisted

Open
#3,685 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

On the pangeo gitter @apatlpo asked how to check if a Dask Array (or any collection) has been persisted. The daskboard provides some indication, but it'd be nice to check programmatically as well.

One option is to check if the task graph backing the collection is "simple", in the sense that there's one task per block / partition:

```python
In [44]: x = da.random.random((100, 100), chunks=10) + 1

In [45]: y = x.persist()

In [46]: len(x.dask) == np.prod(x.numblocks)
Out[46]: False

In [47]: len(y.dask) == np.prod(y.numblocks)
Out[47]: True
```

There are probably cases where this heuristic fails though. An alternative is to ask the scheduler whether it has all the keys of the collection in memory. That would probably be more robust.

If we have a robust way of checking, is this worth adding method to make this easier to check? Perhaps `Client.is_persisted(collection)`?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.