dask / dask/distributed

Distributed compute doesn't appear to respect resource requirements

Open
#3,445 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I have attempting to ensure that only 1 particular key requires the usage of GPU, while the rest should be scheduled fairly over all other systems.

```py
import dask
import dask.distributed

def inc(x):
worker = dask.distributed.get_worker()
print(f'inc {x}')
print(worker.name)
print(worker.total_resources)
print(worker.available_resources)
return x + 1

def add(x, y):
worker = dask.distributed.get_worker()
print(f'add {x} {y}')
print(worker.name)
print(worker.total_resources)
print(worker.available_resources)
return x + y

def double(x):
worker = dask.distributed.get_worker()
print(f'double {x}')
print(worker.name)
print(worker.total_resources)
print(worker.available_resources)
return x * 2

def main():

a = dask.delayed(inc)(1)
b = dask.delayed(inc)(2)
c = dask.delayed(add)(a, b)
d = dask.delayed(double)(c)

with dask.distributed.Client('127.0.0.1:3201', set_as_default=False) as scheduler:
scheduler_info = scheduler.scheduler_info()
print('Scheduler: %s', scheduler_info['address'])
print('Status Port: %s', scheduler_info['services']['dashboard'])

future = scheduler.compute({
'a': a,
'b': b,
'c': c,
'd': d
}, resources={
'c': {
'CPU': 50
},
'd': {
'GPU': 1
}
})

result = future.result()

print('RESULT')
print(result)

input()

if __name__ == '__main__':
main()
```

When I run the above, I see that the `a`, `c` and `d` gets executed on the worker with `GPU=1`, but `b` gets executed on the worker with `CPU=100`. Note that I used `CPU=...` as a just a dummy resource to see what would happen. Most likely I will have workers that have no resource specifications, and only 1 worker process with `GPU=1`.

It seems like the resource constraints is not being obeyed explicitly. Is this how it is meant to be used.

In the above code, I would have expected that `d` gets computed on the GPU worker, but `a`, `b`, and `c` be executed on the CPU workers.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.