Worker profiling may return inaccurate results if event loop is busy.
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
`Worker.trigger_profile` is [currently implemented](https://github.com/dask/distributed/blob/6d85a8536ee02fd2a9ae52f50de030d1cc7afe7e/distributed/worker.py#L878-L886) with `PeriodicCallback`. Given that we do not seem to associate single profiling intervals with timestamps, this means that profiling could be skewed if the loop is busy since we do not profile whatever is hogging the loop and the next iteration of profiling may be pushed back.
`profile.watch` on the other hand [is implemented](https://github.com/dask/distributed/blob/6d85a8536ee02fd2a9ae52f50de030d1cc7afe7e/distributed/profile.py#L387-L396) using a dedicated thread.
This may not be a problem in practice, but we should be aware of this. The skew is somewhat mitigated by profile cycling, which creates tumbling windows and thus isolates skew into individual windows.
Contributor guide
Assessment
This issue has not been assessed yet.