`Adaptive` should update currently used target values on scheduler
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
`Adaptive` can be initialized with multiple options that impact the calculation of desired number of workers.
see https://github.com/dask/distributed/blob/9d8e18c8bcc90f3f4677a41b1d5c89119dd21d17/distributed/deploy/adaptive.py#L85-L108
this works as expected since the `Scheduler.adaptive_target` function is called as an RPC from `Adaptive` with whatever value is used.
However, whenever `Scheduler.adaptive_target` is called outside of the context of the `Adaptive` class, this call will not reflect the currently / most recently used values.
Notable calls to this API outside of the `Adaptive` class
- HTTP API `/api/v1/adaptive_target` (which does not even accept any request paramters)
- HTTP API `/counts/json` which exposes this value as well
- Prometheus metrics `dask_scheduler_desired_workers`
Ideally, a freshly created Adaptive class has the possibility to tell the scheduler what the correct paramters for the adaptive_target are s.t. these observability APIs can output the proper value
Contributor guide
Assessment
This issue has not been assessed yet.