dask / dask/distributed

MemoryState is taking non trivial amount of CPU time in dashboard

Open
#8,726 0 comments 0 reactions 0 assignees View on GitHub
needs triage
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I did some profiling on larger clusters (see also https://github.com/dask/distributed/pull/8725) and noticed the construction of [`MemoryState`](https://github.com/fjetter/distributed/blob/4c426a545e35c03acc6533b81a1ec8aa3ad7a664/distributed/scheduler.py#L262-L311) to take a non-trivial amount of time despite it being defined as using slots.

This shows up particularly in places where the full state object is not required, e.g.
- https://github.com/fjetter/distributed/blob/4c426a545e35c03acc6533b81a1ec8aa3ad7a664/distributed/dashboard/components/scheduler.py#L394
- https://github.com/fjetter/distributed/blob/4c426a545e35c03acc6533b81a1ec8aa3ad7a664/distributed/dashboard/components/scheduler.py#L4258

In https://github.com/fjetter/distributed/blob/4c426a545e35c03acc6533b81a1ec8aa3ad7a664/distributed/scheduler.py#L1849 we are creating possibly hundreds of those objects just to throw them away right away.

I think if we wanted to maintain such a MemoryState object we should ensure that the instances are being updated instead of recreating them on demand given that those dashboards update every 100ms.

Overall this is just 3% of CPU time in my measurement (about the same size as https://github.com/dask/distributed/pull/8725) so it is not urgent.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.