Investigate performance issues in profiler
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
This is a bit vague, apologies. We've received a report of high memory (multi GiB) usage that was resolved by reducing the frequency of the profiler in the workers. The user used tracemalloc and noted that many large allocations were occurring in the profiler. Reducing the frequency of profile info collections reduced the memory usage significantly.
See https://github.com/PrefectHQ/prefect/issues/3238 and https://stackoverflow.com/questions/63680134/growing-memory-usage-leak-in-dask-distributed-profiler/63680548#63680548
Other related issues also seem to point to performance overhead in the profiler
- https://github.com/dask/distributed/issues/4035
- https://github.com/dask/distributed/issues/2156#issuecomment-503735503
Contributor guide
Assessment
This issue has not been assessed yet.