dask / dask/distributed

Worker profiling may return inaccurate results if event loop is busy.

Open
#6,499 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

`Worker.trigger_profile` is [currently implemented](https://github.com/dask/distributed/blob/6d85a8536ee02fd2a9ae52f50de030d1cc7afe7e/distributed/worker.py#L878-L886) with `PeriodicCallback`. Given that we do not seem to associate single profiling intervals with timestamps, this means that profiling could be skewed if the loop is busy since we do not profile whatever is hogging the loop and the next iteration of profiling may be pushed back.

`profile.watch` on the other hand [is implemented](https://github.com/dask/distributed/blob/6d85a8536ee02fd2a9ae52f50de030d1cc7afe7e/distributed/profile.py#L387-L396) using a dedicated thread.

This may not be a problem in practice, but we should be aware of this. The skew is somewhat mitigated by profile cycling, which creates tumbling windows and thus isolates skew into individual windows.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.