Worker profile limited to a short timespan
- 主要言語
- Python
- スター
- 1.7k
- フォーク
- 778
- 平均マージ
- 2時間 50分
- マージ済み PR(30日)
- 3
説明
**Describe the issue**:
The worker profile has a limited span and older data seems to be lost. For example, with the minimal example below, the total CPU time is 24 hours, but the profile never contains more than 4 h 26 min. At one point during the run, the profile looks like this:

30 minutes later, it looks like this:

The previous data is completely gone, as can be seen by the "activity over time" graph at the bottom.
This issue has been occurring for several months, most recently with dask and distributed 2024.4.2. You can look at the full discussion on Discourse: https://dask.discourse.group/t/measuring-the-overall-profile-of-long-runs/1859/11
**Minimal Complete Verifiable Example**:
```python
import time
import logging
from dask.distributed import (
Client,
LocalCluster,
get_client,
as_completed,
performance_report,
)
NUM_PROCESSES = 16
logging.basicConfig(
level=logging.DEBUG,
format="%(asctime)s %(levelname)-8s %(message)s",
)
class DummyManager:
def run(self):
logging.info("Starting the manager")
jobs = list(range(1, 97))
client = get_client()
futs = []
for j in jobs:
futs.append(client.submit(self.job, j))
asc = as_completed(futs, with_results=True)
for fut, ret in asc:
logging.info(f"Processing future {str(fut)}: ret={str(ret)}")
if ret > 0:
logging.info(f"Launching a subjob with time {ret}")
asc.add(client.submit(self.job, ret))
fut.release()
def job(self, n):
time.sleep(60 * 15)
return 0
if __name__ == "__main__":
cluster = LocalCluster(
n_workers=1,
threads_per_worker=NUM_PROCESSES,
processes=False,
)
client = Client(cluster)
with performance_report(filename=f"dask-performance_{time.time():.0f}.html"):
try:
manager = DummyManager()
manager.run()
except KeyboardInterrupt:
logging.info("Stopping the job...")
cluster.close()
exit(0)
client.close()
cluster.close()
```
**Environment**:
- Dask version: 2024.4.2
- Python version: 3.10.12
- Operating System: Linux Mint 21.2
- Install method (conda, pip, source): pip
コントリビューションガイド
調査の方向性
Run the provided Python minimal example with Client, LocalCluster, and performance_report, then compare the worker profile and activity-over-time view during the long run. Trace how profiling data is collected and retained, using the linked Discourse discussion for context. Done means older activity remains available and the profile covers the full run rather than stopping around 4 h 26 min.
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- python
- 領域
- distributed-systems, observability-sre
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 停滞
- 明瞭さ
- 説明が足りない
- 初心者へのやさしさ
- 35/100