Exceeding cluster_max_cores error message
- Dominant language
- Python
- Stars
- 148
- Forks
- 93
- PR merge metrics
- No merged PRs in 30d
Description
Hello, Jim.
It may be helpful to clarify whether or not the resource usage of schedulers contributes to the cluster limits. For example, if `cluster_max_cores = 10` and the scheduler uses 0.5 cores, can a user spin up 10 x 1-core workers or just 9? I think that the answer is 9 based on some testing that I conducted, but I didn't see anything in the documentation that clarified this distinction. I may have just missed it though. Happy to submit this documentation PR if it makes life easier for you.
When I tested the above scenario (`cluster_max_cores = 2` and attempting to scale to 2 x 1-core workers), I ran into the following error message:
```python
/opt/anaconda3/envs/dask/lib/python3.7/site-packages/dask_gateway/client.py:668: GatewayWarning: Scale request of 2 workers would exceed resource limit of 1 workers. Scaling to 1 instead.
warnings.warn(GatewayWarning(msg["msg"]))
```
This message can be misleading because it gives the impression that the problem is with the `cluster_max_workers` configuration, when in fact, I had that set to `100`. I suspect that exceeding the `cluster_max_memory` returns a similar error message. It might be worth making this error message more informative.
Curious to hear your thoughts.
**Environment**:
Gateway Server: 0.8.0
Gateway Client: 0.8.0
Python: 3.7.4
Dask: 2.22.0
Distributed: 2.22.0
Contributor guide
Assessment
This issue has not been assessed yet.