Negative occupancy causes dashboard and metrics to fail
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
```python-traceback
(scheduler) 2023-02-10 23:33:02,530 - tornado.application - ERROR - Uncaught exception GET /metrics (10.0.21.121)
HTTPServerRequest(protocol='http', host='10.0.21.121:8787', method='GET', uri='/metrics', version='HTTP/1.1', remote_ip='10.0.21.121')
Traceback (most recent call last):
File "/opt/coiled/env/lib/python3.9/site-packages/tornado/web.py", line 1711, in _execute
result = method(*self.path_args, **self.path_kwargs)
File "/opt/coiled/env/lib/python3.9/site-packages/distributed/http/scheduler/prometheus/core.py", line 184, in get
self.write(prometheus_client.generate_latest())
File "/opt/coiled/env/lib/python3.9/site-packages/prometheus_client/exposition.py", line 198, in generate_latest
for metric in registry.collect():
File "/opt/coiled/env/lib/python3.9/site-packages/prometheus_client/registry.py", line 97, in collect
yield from collector.collect()
File "/opt/coiled/env/lib/python3.9/site-packages/distributed/http/scheduler/prometheus/core.py", line 34, in collect
value=self.server.adaptive_target(),
File "/opt/coiled/env/lib/python3.9/site-packages/distributed/scheduler.py", line 7777, in adaptive_target
(self.total_occupancy + queued_occupancy) / target_duration
File "/opt/coiled/env/lib/python3.9/site-packages/distributed/scheduler.py", line 1771, in total_occupancy
return self._calc_occupancy(
File "/opt/coiled/env/lib/python3.9/site-packages/distributed/scheduler.py", line 1794, in _calc_occupancy
assert occ >= 0, occ
AssertionError: -39340.42575940865
```
This cluster already appeared to be quite broken, so I imagine some other state was already off (i.e. this is a symptom of some other problem).
Having this fail in both the metrics and the dashboard endpoints is unfortunate; it makes you completely blind to whatever else is going on.
I could probably reproduce this, but certainly not minimally yet. Opening to note that negative occupancies do seem to be able to occur, and to track in case it happens to anyone else.
Contributor guide
Assessment
This issue has not been assessed yet.