dask / dask/distributed

Query collections that are currently persisted

Open
#7,080 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

People often run into situations where they have excess data in memory and don't know why. A common culprit for this is persisted collections or futures sitting in Jupyter state. It's hard for users to become aware of this lingering state.

Advanced users could look at the futures owned by a client, but this isn't super-accessible, and it's not always easy to back out larger collections from these futures.

Some possible thoughts:

1. Futures could have weak references to collections that contain them (we would populate this during persist calls). Clients could then have a `client.collections_in_memory` method or whatever
2. The scheduler could have dashboards showing high level structures that are in RAM better. We already have this in the various progress and TaskGroup plots but it's subtle (we show groups or prefixes that are in memory by making them less transparent than otherwise). This wouldn't show collections but rather groups, but maybe that's ok.

cc @phobson who ran into this recently

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.