dask / dask/distributed

`Adaptive` should update currently used target values on scheduler

Open
#7,451 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

`Adaptive` can be initialized with multiple options that impact the calculation of desired number of workers.

see https://github.com/dask/distributed/blob/9d8e18c8bcc90f3f4677a41b1d5c89119dd21d17/distributed/deploy/adaptive.py#L85-L108

this works as expected since the `Scheduler.adaptive_target` function is called as an RPC from `Adaptive` with whatever value is used.

However, whenever `Scheduler.adaptive_target` is called outside of the context of the `Adaptive` class, this call will not reflect the currently / most recently used values.

Notable calls to this API outside of the `Adaptive` class

- HTTP API `/api/v1/adaptive_target` (which does not even accept any request paramters)
- HTTP API `/counts/json` which exposes this value as well
- Prometheus metrics `dask_scheduler_desired_workers`

Ideally, a freshly created Adaptive class has the possibility to tell the scheduler what the correct paramters for the adaptive_target are s.t. these observability APIs can output the proper value

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.