Documentation / API to check if a Dask collection is persisted
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
On the pangeo gitter @apatlpo asked how to check if a Dask Array (or any collection) has been persisted. The daskboard provides some indication, but it'd be nice to check programmatically as well.
One option is to check if the task graph backing the collection is "simple", in the sense that there's one task per block / partition:
```python
In [44]: x = da.random.random((100, 100), chunks=10) + 1
In [45]: y = x.persist()
In [46]: len(x.dask) == np.prod(x.numblocks)
Out[46]: False
In [47]: len(y.dask) == np.prod(y.numblocks)
Out[47]: True
```
There are probably cases where this heuristic fails though. An alternative is to ask the scheduler whether it has all the keys of the collection in memory. That would probably be more robust.
If we have a robust way of checking, is this worth adding method to make this easier to check? Perhaps `Client.is_persisted(collection)`?
Contributor guide
Assessment
This issue has not been assessed yet.