common-workflow-language / common-workflow-language/cwltool
[Proposal] customizable Job executor
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
Hello! I would like to propose a design for an extension to cwltool that allows the user to plug in customized job executors in addition to the existing ones (shell, docker).
The rationale for this is to integrate with services that provide code execution, like kubernetes, aws ec2, or some APIs internal to a company.
There are two possible ways to design this, in my opinion:
- Allowing to pass in a path to a python module that contains a subclass of `JobBase` that will be used instead of the existing ones. An instance of this job class will be returned by `CommandLineTool.makeJobRunner`.
- Implementing a new subclass of `JobBase`, called eg `ExternalJob`, that will be shipped inside `cwltool`, then allowing the user to pass in a module that will contain some methods that can be used by the `ExternalJob` class to start a job and check if a job is completed. ExternalJob will pass to the method all the required parameters to start a job (docker image, volumes, cwd, env vars, etc).
The advantage of the first approach is that it provides the greatest flexibility in terms of implementation. The advantage of the second approach is that it's higher level, so a lot of details are taken care of by the `ExternalJob` and the third party code would be easier to write.
I already have a POC for the second approach, and I would be very happy to collaborate to have such a mechanism being pushed upstream. I would understand if this is behind the scope of cwltool, but I personally think this is a very powerful addition.
cheers ✌️
Contributor guide
Assessment
This issue has not been assessed yet.