common-workflow-language / common-workflow-language/cwltool
Allocation of CUDA devices not retrieved correctly
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
## Expected behavior
When running a workflow where a step uses the "cwltool:CUDARequirement" requirement, the run should be allocating the required number of CUDA devices to each job. When using scattering with "--parallel", new jobs for that step should only be spawned if a sufficient number of unallocated CUDA devices is available.
## Current behavior
New jobs are spawned irrespective of the current number of allocated CUDA devices. This leads to multiple jobs being spawned with a single CUDA device and failing jobs due to insufficient resources.
Using "--debug", pending jobs, e.g. due to insufficient RAM, always return "allocated CUDA is 0" even though CUDA devices should be allocated (debug output like 'Job "XXX" cannot be run, requests more resources (XXX) than available on this host (already allocated ram is XXX, allocated cores is XXX, allocated CUDA is 0, max ram XXX, max cores XXX, max CUDA XXX).')
## Possible solutions
The Problem was fixed on my machine by editing line 418 in executors.py (run_job function):
`cuda = cast(int, job.builder.resources.get("cudaDevices", 0))`
with
`cuda = cast(int, job.builder.resources.get("cudaDeviceCount", 0))`
as "cudaDevices" does not seem to be the correct name of the resource.
Contributor guide
Research direction
Start in executors.py at the run_job function around line 418, where the issue identifies the CUDA resource lookup. Check the existing resource-allocation behavior for scattered jobs with --parallel and the --debug message for pending jobs. Done means jobs wait for sufficient unallocated CUDA devices and debug output reports the allocated CUDA count correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, cli
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100