dask / dask/distributed

Negative occupancy causes dashboard and metrics to fail

Open
#7,538 6 comments 0 reactions 0 assignees View on GitHub
bug needs info scheduler
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

```python-traceback
(scheduler) 2023-02-10 23:33:02,530 - tornado.application - ERROR - Uncaught exception GET /metrics (10.0.21.121)
HTTPServerRequest(protocol='http', host='10.0.21.121:8787', method='GET', uri='/metrics', version='HTTP/1.1', remote_ip='10.0.21.121')
Traceback (most recent call last):
File "/opt/coiled/env/lib/python3.9/site-packages/tornado/web.py", line 1711, in _execute
result = method(*self.path_args, **self.path_kwargs)
File "/opt/coiled/env/lib/python3.9/site-packages/distributed/http/scheduler/prometheus/core.py", line 184, in get
self.write(prometheus_client.generate_latest())
File "/opt/coiled/env/lib/python3.9/site-packages/prometheus_client/exposition.py", line 198, in generate_latest
for metric in registry.collect():
File "/opt/coiled/env/lib/python3.9/site-packages/prometheus_client/registry.py", line 97, in collect
yield from collector.collect()
File "/opt/coiled/env/lib/python3.9/site-packages/distributed/http/scheduler/prometheus/core.py", line 34, in collect
value=self.server.adaptive_target(),
File "/opt/coiled/env/lib/python3.9/site-packages/distributed/scheduler.py", line 7777, in adaptive_target
(self.total_occupancy + queued_occupancy) / target_duration
File "/opt/coiled/env/lib/python3.9/site-packages/distributed/scheduler.py", line 1771, in total_occupancy
return self._calc_occupancy(
File "/opt/coiled/env/lib/python3.9/site-packages/distributed/scheduler.py", line 1794, in _calc_occupancy
assert occ >= 0, occ
AssertionError: -39340.42575940865
```

This cluster already appeared to be quite broken, so I imagine some other state was already off (i.e. this is a symptom of some other problem).

Having this fail in both the metrics and the dashboard endpoints is unfortunate; it makes you completely blind to whatever else is going on.

I could probably reproduce this, but certainly not minimally yet. Opening to note that negative occupancies do seem to be able to occur, and to track in case it happens to anyone else.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.