dask / dask/distributed

Investigate performance issues in profiler

Open
#4,091 2 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

This is a bit vague, apologies. We've received a report of high memory (multi GiB) usage that was resolved by reducing the frequency of the profiler in the workers. The user used tracemalloc and noted that many large allocations were occurring in the profiler. Reducing the frequency of profile info collections reduced the memory usage significantly.

See https://github.com/PrefectHQ/prefect/issues/3238 and https://stackoverflow.com/questions/63680134/growing-memory-usage-leak-in-dask-distributed-profiler/63680548#63680548

Other related issues also seem to point to performance overhead in the profiler

- https://github.com/dask/distributed/issues/4035
- https://github.com/dask/distributed/issues/2156#issuecomment-503735503

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.