dask / dask/distributed

Worker profile limited to a short timespan

未关闭
#8,653 3 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
bug
主要语言
Python
星标
1.7k
派生
778
平均合并
2 小时 50 分钟
30 天内合并 PR
3

描述

**Describe the issue**:
The worker profile has a limited span and older data seems to be lost. For example, with the minimal example below, the total CPU time is 24 hours, but the profile never contains more than 4 h 26 min. At one point during the run, the profile looks like this:
![image](https://github.com/dask/distributed/assets/52597883/532af79a-0158-4035-91f6-fa76f663229c)

30 minutes later, it looks like this:
![image](https://github.com/dask/distributed/assets/52597883/37702c5a-e092-4b4a-8572-ddfb17d3a209)

The previous data is completely gone, as can be seen by the "activity over time" graph at the bottom.

This issue has been occurring for several months, most recently with dask and distributed 2024.4.2. You can look at the full discussion on Discourse: https://dask.discourse.group/t/measuring-the-overall-profile-of-long-runs/1859/11

**Minimal Complete Verifiable Example**:

```python
import time
import logging

from dask.distributed import (
Client,
LocalCluster,
get_client,
as_completed,
performance_report,
)

NUM_PROCESSES = 16

logging.basicConfig(
level=logging.DEBUG,
format="%(asctime)s %(levelname)-8s %(message)s",
)

class DummyManager:
def run(self):
logging.info("Starting the manager")

jobs = list(range(1, 97))

client = get_client()
futs = []
for j in jobs:
futs.append(client.submit(self.job, j))

asc = as_completed(futs, with_results=True)
for fut, ret in asc:
logging.info(f"Processing future {str(fut)}: ret={str(ret)}")
if ret > 0:
logging.info(f"Launching a subjob with time {ret}")
asc.add(client.submit(self.job, ret))
fut.release()

def job(self, n):
time.sleep(60 * 15)

return 0

if __name__ == "__main__":
cluster = LocalCluster(
n_workers=1,
threads_per_worker=NUM_PROCESSES,
processes=False,
)

client = Client(cluster)
with performance_report(filename=f"dask-performance_{time.time():.0f}.html"):
try:
manager = DummyManager()
manager.run()
except KeyboardInterrupt:
logging.info("Stopping the job...")
cluster.close()
exit(0)
client.close()
cluster.close()
```

**Environment**:

- Dask version: 2024.4.2
- Python version: 3.10.12
- Operating System: Linux Mint 21.2
- Install method (conda, pip, source): pip

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。